Stability fixes

Bug-fix: missing tag_options column

This update fixes an issue for posts, which consist of links only, which are deleted in batch. The resulting post is an empty text (or a text containing new lines and tabs only), and XenForo fails to save this post back to the DB. The update makes sure, that, in case of empty post content the post gets deleted itself, instead of trying to save it with an empty message.

In this version we have implemented attachment validity check. Attachments added to post text can now be checked and will be reported as not found if the attachment with that ID does not exist any longer.

In this version we have also fixed the bug with "403 Forbidden" links be considered as valid (as designed) but appearing in Moderation Queue as invalid. On installation the status of these links will be changed to "449 Retry With", and using Admin -> Tools -> Links -> Rebuild Data page you can targetly re-check these URLs, which will validate them again and remove them from the moderation queue. Just make sure to uncheck the checkbox "Enable Local Cache" to ensure the URLs are re-checked.

Another bug fixed is that in Rebuild Data page unchecking the checkbox "Enable Local Cache" had no effect and the cache was still used.

This release implements cookie support for remote request, which solves the issues with remote servers, that put cookies during the first request to their server and require the agent to accept the cookies, otherwise an infinite redirect to the same URL happens. This was implemented to fix the issue with box.com URLs.

With this version we implemented additional protective measures against server overload. Now server load will be checked before each link is checked to prevent the overload.

We have also improved link status checking algorithm, which will make it much faster especially the check for big files.

This is a bug-fix version solving two issues regarding invalid URL redirect:

1. URLs with status 403 are shown as valid but don't redirect
2. URL redirects are not active if the user does not have permission to see the status of the link

This is a bug-fix release, that solves the issue of deleting the links from Moderation Queue

With this release, we have implemented link reporting functionality. Now users can report a link as invalid, and admins will be able to filter/search links by number of reports links got. Some additional options are implemented in product's settings page to control, if link status icon should be shown or no and if the report link should be shown or no. By default, the icons are shown but report link is disabled.

The version implements minor design changes in invalid URL redirection page and adds a button to the home page.

This version implements an option to redirect all broken URLs to a page, that asks the user confirm if they want to follow the link.

This is a bug-fix release, that solves the wrong 405 response code for some servers, that don't support HEAD request type. For these servers the request will also be sent using traditional GET method.

In this version, we have fixed a PHP error which was happening when multiple posts from the same thread were deleted, including the first post of the thread.

We have also implemented two new options to disable the number of links shown for each status code in admin panel as well as to disable the preview of links. These features make admin panel pages slower in case of large link database.

This version enhanced the control over background tasks scheduled and allows to re-start a task if it was lost for some reason (e.g., the server was restarted while the task was running). The delayed checks (scheduled to check the links that respond with "429" header) can now be canceled/deleted as well. Deleyed tasks are usually registered to run with some delay in between and should not affect board performance, but if you notice any issue with them you can just delete them, and later check them anytime by selecting "429" status in Dead Link Check section.

This version fixes a bug with the message about background tasks disappearing sometimes. There is also a minor phrase fix.

This version improves product's performance of checking link statuses. Instead of using XenForo built-in client we have built our implementation using CURL making sure, that the information of the URL itself does not download to your server, but only the headers of the page are requested. The client tries first "HEAD" type of request, which does not get the body of the page but only headers. Unfortunately, some servers block this type of requests, so in this case, we also send GET type of request. However, we make sure the entire content of the file is not downloaded by canceling the download request as soon as we have the headers available.

The optimization implemented because especially important by the implementation of image support in our product, as some images could be quite big, and fetching them fully into memory could cause memory issues.

There is also a small change in user interface introduced in this version. The list of links on Rebuild Data page is moved under "Check now" button, so it does not disturb admins, who are not in seeing the list and just want to check the status of the links.

This version implements options to disable Moderation Queue and Alert features implemented. Moderation Queue should be disabled before large checks, as it may cause thousands of entries to be added to the queue, which may make the site slower for visiting moderators.

We have also fixed a bug when Cancel button was showing for server load information message.

This version improves product's performance significantly.

In one of the recent versions, we have implemented cleaning up the database from URLs, that are not used in any post. The query checking for such links was missing an index and was executing slower than in should. This process was called on Batch Update page, every time a batch of links is processed. This caused Batch Update page not to respond and increase server load. Now the cleanup process is optimized and runs almost instantly, and it is moved to an hourly cron job.

In this version, we have also introduced server load-based control of background tasks. A new option is available to set the server load limit, above which indexation/dead link check/link update tasks will stop running. If the tasks run in the background (Cron mode), they will re-try every 1 minute and will continue automatically only if server load is normalized. If running in a browser window, a message will be shown including current server load and the page will reload every 15 seconds and will retry to continue the process.

Current server load information and the configured limit are shown to admins on top of Batch Update and Rebuild Data pages.

This is another stability fix release. All reported PHP errors are fixed and the compatibility with Trending Content Tag Addon by @Xon is fixed. We have also changed the default tag type to All.

This is another big-fix release improving add-on stability of under heavy load. Now all tag-related database queries will run considering possible table lock in another process.

There was a bug preventing link index update when threads are deleted. This bug is now fixed and threads deleted after the upgrade will be re-indexed.

There were several small fixes and enhancements as well, like choosing "Delete tag" as default option when there are no other options, selecting "Replace fully" or "Replace found part" in replacement options as soon as you start typing the replacements. Now "Delete tag" will also allow to set the replacement option even if there is no content type and tag type selected.

This is a bug-fix release with improved stability in case of unexpected URL structures or table lock in the database. We have also implemented "Cancel" button for ongoing background tasks, which will allow you to stop the ongoing background task, both for batch updates and dead link checks.

This version fixes the bug with deleting tags from Admin Panel.

Hello, everyone!

We are glad to announce the release of the next version of Link Checker, which is a complete rewrite of the product with many new features implemented.

Please check the main add-on description for detailed information on how the add-on works.

Here is a quick overview of features implemented in this version:

1. Support for three types of BB Code tags - URL, MEDIA and IMG. Supported media types are YouTube, DailyMotion, Vimeo. Other media types can be supported if requested.
2. A content-specific search of links - for now, posts are the only content type supported, and it is possible to find all links filtering by post date, thread date, and thread ID.
3. Tag-specific search and replacement options. The Batch Update page has dynamic interface updating every time the type of tag is changed, providing options specific for that tag.
4. Deletion of the content (posts or threads) instead of deleting the links.
5. Cron-based batch update and URL validation. Batch Update and Rebuild Data pages will show information about the ongoing background tasks.
6. Improved domain-based filters in Batch Update and Rebuild Data pages.
7. Checkboxes in Batch Update and Rebuild Data pages allowing admins to select the links they want to process, without affecting any other links found.
8. New front-end feature - Alert the owner of the content if their link (URL, media or image) is invalid. The alert is removed automatically if the link is re-checked and its status is updated to valid. Controlled by a new usergroup permission.
9. New front-end feature - Add invalid links to the moderation queue, allowing moderators to edit them and save or delete them. Moderators in particular forums will only see links in that forums. The moderation entry is removed automatically if the link is re-checked and its status is updated to valid. Moderator actions are logged in XenForo Moderator Log, showing the text of the link before and after moderation. Controlled by a new usergroup permission.

Here is a video demonstrating the new Batch Update page interface with an example of Regular Expressions usage.

Important upgrade note: This is a complete rewrite of the product, so many files and the database structure are updated. To avoid any errors during installation, please disable the add-on before uploading add-on files. Import the XML provided in the package and only after that enable the add-on. The installation runs migration of the data from the old structure to the new. This may take too long on board with millions of links already in the database and the installation script may timeout. In this case, it is recommended to uninstall the product and install the new version after which you will need to run the indexation and Dead link validation one more time.

With this version, we are changing the price for the add-on to $34.99 USD with $25.00 USD yearly renewal fee. The change does not apply to our existing customers.

Here is a minor update that implements a new option to show link status as a tooltip on the icons in front-end.

In this release, we have improved the algorithm, especially fixing the cases, when the remote server denies the requests and replies with "429 Too Many Requests" header. Now the product will delay the check to such servers automatically, checking the links every 1 second in secondary (deferred) tasks.

On Rebuild Data page there is a checkbox now to check only URLs that failed previously with "429" header. Once the check is complete, you will see the number of deferred tasks and when the last one is scheduled to execute. Visiting Rebuild Data page, you will see this information if there are any deferred tasks. Once there are no deferred tasks left, you can see link statuses in Batch Update page, there should not be any 429 status code anymore.

Another bug was causing valid URLs to be shown as invalid on front-end. This was the case only when the same page has two or more of the same URL, but the check for the URL was successful one time, and it failed some other time. Now, if the link was checked successfully at least one time, its status will be valid on front-end.

This is a minor release with some new UI features and stability enhancements.

After Batch Processing the message will show how many links are updated/deleted. The same page will contain buttons to go back to the Batch Update or Manage Links pages.

After the tasks on Rebuild Data page run, they will show the message "Cache rebuild successfully"

We have got a report about a link returning error "Error in cURL request: Empty reply from server". Our product sends so called "head" request to check link statuses - it does not fetch the entire content of the page, but only asks for headers. This makes the requests faster and saves bandwidth. Some servers, however, don't seem to support this type of request. We have implemented a workaround for these cases, and for such servers full request will be sent.

We have received some reports, that some links don't work in our product, but works in the browser. We have troubleshot the cases and see, that sometimes they are caused by "User Agent" of the product being blocked on the remote site. By default, the "User Agent" contains your board URL so that the remote site may have blocked your site by URL. Our later versions changed the user agent to "AddonsLab/URLStatusChecker/1.0", however, it seems, the word "Checker" is blocked on some sites.

So we have decided to make an option that allows you set the user agent to use. This does not, however, mean, that you should change the user agent often to avoid being blocked by remote sites. Doing so may even be illegal in some cases. Please choose an identifier that works the best for you, and do not change it. Respect the right of remote sites to block your site if they don't want you to send requests to them.

User agent format is usually ProductName/number.number (any comment here).

Here is a bug-fix release, that solves the problem of search triggered when all criteria in Batch Update page are empty.

With this release, we have implemented several enhancements.

Now Batch Update page will keep the values entered even if you visit the page later (stored in cookies). After deleting a link from Quick Edit form, it will redirect back to Batch Update page, and the page will already be pre-filled with the values you had there.

We have also implemented a way to convert a URL into a text by setting the "Process URL" field to Replace Fully and leave the field empty. Link Text will be used as a replacement.

A new option is added to define the number of links to show in the preview in Batch Update page. Another field is now available in the Batch Update page, where you can set the maximum number of links to process. You can use it if you are not sure how the replacement will work exactly so that you can set a minimum limit and check manually how the replacement worked. You can also change the default limit of 1000 in case processing of so many links times out on your server setup.

This release fixes a bug with Action Log page not showing Replacement text properly if a link is deleted from Batch Update page.

This release fixes the bug with status messge not fitting in last_status_message column in some cases.

This is a bug-fix release, that solves the problem of batch deletion/updating being limited to 40 links only.

Delete Action Log button is implemented.

Here is a bug-fix release of the product, that fixes some stability issues, because of which the product was showing PHP errors in some cases. Particularly, malformed URL could have the domain part to exceed the allowed limit of 255 characters, which was causing DB error. Now a domain name longer than 255 characters is considered as invalid immediately. We have also fixed the case when a none-Unicode message was saved to the database, because of the URL having none-Unicode characters in it, and this caused a DB error. Now the message will be cleaned-up before being sent to the database.

Here is a bug-fix version, that prevents Dead Lock errors, in case two processes try to rebuild the same post at the same moment. It also makes it possible to remove link text in Batch Update mode, by choosing "Replace Fully" in "Process Text" section and leaving the text field empty.

In this version, we have made some minor enhancements and fixes in the user interface. The Check Status link on Edit Link page is removed, as it was checking the status based on old link URL, not the edited one, and was misleading. Now link status will be rechecked and updated every time a link is saved. To allow administrators to quickly skip validation of any link (e.g. link gives an error but you still want to use it), you can choose to Skip Validation from Edit Link page. Please note, that this option does not replace the Exclusion List, it will skip the validation for that particular link only in that particular post.

Another enhancement/fix introduced in this version is storing the current filter in Manage Links page constantly, even when you navigate to another page and come back later. The filters activated will stay active unless you clear them and submit the form again.

We have also implemented an additional filter in Batch Update page - External domains only. If checked, only links not belonging to your site's domain will be shown and processed.

We are also introducing a new option - Disable Front-End Features. If front-end features are disabled no additional resources will be used on front-end, link information will not be loaded, link icons and Check Links will not be available, no additional JavaScript file will be loaded.

We have got a report, that all indexed links were deleted from the database without any action from admin's side. This did not affect the links in the posts themselves, but only link index was removed. After an intensive investigation, we could not reproduce the issue, but we found some edge-cases when this could happen if the data expected by default XenForo installation/setup is not available, possibly because of another add-on using and modifying the same data. We have added additional safety measurements to the product, to make sure these edge-cases don't affect product functionality and the links indexed are not removed.

Re-indexation of links from Rebuild Data page and checking link status from the same page will restore the information in the database. The process should be much faster compared to the initial check, as all links already checked have their information in the database and there should be no need for any remote requests.

This bug-fix release fixes a logical bug in the way exclusion list works. Now it will work as expected, if any word/snippet provided in the exclusion list is found in the URL the URL will be considered as valid.

Here is a bugfix release solving several issues reported recently. Clicking Check Links link under posts was not fixing the status automatically, now the status will be stored in the database properly and the icon will update immediately.

Deleting posts was deleting the links associated with it, but deletion an entire thread was leaving the links in the database. This is now solved as well.

We have also fixed some edge cases of URL detection.

Here is a minor update giving admins an option to exclude some URLs from the remote check. A new option is added in product's Options page, where the excluded URLs should be specified, one per line. Any snippet specified is checked to match any part of the URL so that you can specify the domain, the full URL or any other part of the URL there. Any URL having any of these snippets in it will be ignored and considered as a valid URL.

The main reason we have implemented this functionality is that we are getting reports from our customers, which several URLs are actually alive but the product does not recognize it and gives an error when trying to save the URL in Admin Panel.

It is important to realize, that checking a URL automatically is not the same as visiting the URL in your browsers. There are many techniques available, that can make a page to work when visiting by the browser, or when Google indexes the site, but block any other bots, including our bot that checks the status of URL.

Feel free to contact us whenever you get any URL that doesn't work for you in our add-on, but works via browser, but so far in all cases we had this resulted from the site/ip being blocked by another site or automated requests being blocked altogether.

This information will be also added to add-on description text for further reference.

This is a minor upgrade with several small changes introduced.

From this version on the links with status "403 Forbidden" will be considered as valid, and will not cause an error when trying update a link with URL, which is not accessible, as there are many URLs, which are valid, just they are not accessible by a guest user.

All requests now will be made from User Agent called "AddonsLab/URLStatusChecker/1.0" instead of auto-generated agent that contains board URL. In upcoming versions we will give admins an option to change the User Agent identifier used on their own.

This also solves a bug when link statuses on in the post are not automatically updated after Check Links is clicked.

This is a bug-fix release, that solves a case with a valid license showing

In this version, we introduce full log of all actions handled via Admin Panel. Both editing/deleting individual links and batch processing links creates a log, which stores how the link was looking before and after editing, both its URL and text. This is to secure admins in case they change something by mistake. They can always find the link and see its original form.

We have also implemented better support for non-URL characters in URLs. The previous version was handling the URLs properly as well, but it could cause confusion for admins seeing the correct characters in their browser, but the URL-encoded version in our Admin Pages.

For all admins working on boards using Unicode and other non-URL compatible characters in URLs we would like to stress the importance of understanding, how URLs containing special characters work. URLs themselves are not allowed to have Unicode/special characters. All Unicode characters are converted to their compatible version (URL encoding). You can confirm this by editing any post, which has a Unicode-containing link and see the source code. And you will see the link does not contain Unicode characters and is fully encoded. Basically, our product just takes the URLs from the post and processes them just as they are in the text. If a user follows the URL, browsers show the text in Unicode characters, but this does not mean, that the URL actually has this characters inside. We have decided, however, to avoid confusion for admins, to show decoded characters on the list and Batch Update page. When you are editing a link, however, we will show the real URL, and any URL you paste as a replacement should be properly encoded. You should not try to type Unicode characters in the field directly, but rather copy the link from a browser, which will automatically do the conversion for you.

Here is the list of processes, where we do URL encoding conversion:

1. In Manage Links page, you will see the Unicode characters inside URLs.
2. When editing a link, you will see the Unicode characters in Full Code sector but will see the encoded version in the field to edit the URL.
3. On Batch update page, Search Link URL content will be automatically URL-encoded before the search. So the text you type there can be in Unicode or contain special characters, they will be converted to URL format on the fly.
4. Process URL field content will be transformed into URL-encoded version only if you use "Replace found part" mode. So, if using "Replace found part" mode, you should use Unicode characters in both search and replacement field. However, if you are using "Replace Fully" mode, the URL you will use in the field should be fully valid, meaning all Unicode characters should be already converted to URL-compatible version.
5. Regular expression matching is done after the search term is URL-encoded.
6. Subpattern replacement is done before the replacement term is converted to URL-encoded version.
7. In replacement preview, you will see the Unicode characters, but in actual replacement will use valid URL-encoded strings.

All admins dealing with Unicode URLs are strongly advised to backup their database before using the tool. We recommend testing batch replacement with one-two URLs only, checking the actual post content, and making sure the replacement was processed as expected.

We are glad to help with advice on how to do the replacement if you are not sure. Just contact us in our Support System and we will look at it asap. Please note, that we are not responsible for any loss or corruption of data as a result of your usage of our add-on so please contact us before running batch update tools for an advice.

Performance optimization in a MySQL query

This is a bug-fix release, that solves the issue with post links not fitting in the cache when there are too many links in one post.

Initial release

• Stability fixes

• Bug-fix: missing tag_options column

• Bug-fix: saving empty post after removal of all links produces a PHP error

• Attachment check
• Bug-fix: 403 Forbidden links shown in the Moderation Queue
• Bug-fix: Unchecking "Enable Local Cache" checkbox does not work

• Cookie support for remote requests

• Performance optimizations and server overload prevention

• URLs with status 403 are shown as valid but don't redirect
• URL redirects are not active if the user does not have permission to see the status of the link

• Bug-fix: Links not deleted when deleting the content from moderation queue

• Report Link functionality
• Option to disable link status icon in the posts
• Option to disable Report link

• Minor design changes in invalid URL redirection page

• New option: Enable Invalid URL Redirect

• Wrong 405 response code for some URLs

• Fixed PHP error
• Admin panel performance optimization options

• Ability to restart a stuck background task
• Ability to delete the delayed checks

• Bug: Background task information disappearing sometimes
• Phrase wrong on Batch Update processing page

• Rewritten remote URL checker with memory optimizations
• Small change of UI in Rebuild Data page

• New option: disable moderation queue
• New option: disable alert notifications
• Bug-fix: do not show cancel button for server load info

• Bug-fix: database clean-up query running too slow
• Server load-based running of the tasks

• Bug-fix: undefined method
• Bug-fix: compatability with Trending Content Tag Addon by @Xon
• Changed default tag type to All

• Transaction lock prevention fix
• Delete thread does not delete thread links from index
• Delete Tag is not selected as default option when there are no other options
• "Delete Tag" not showing the replacement message in some cases

• Stability fixes for invalid URLs
• Cron task cancellation
• Prevention of DB error

• Bug fix: delete link from Admin Panel does not work from

1. Support for three types of BB Code tags - URL, MEDIA and IMG
2. A content-specific search of links
3. Tag-specific search and replacement options
4. Deletion of the content (posts or threads) instead of deleting the links
5. Cron-based batch update and URL validation
6. Improved domain-based filters
7. Checkboxes in Batch Update and Rebuild Data pages allowing admins to select the links they want to process
8. New front-end feature - Alert the owner of the content if their link (URL, media or image) is invalid
9. New front-end feature - Add invalid links to the moderation queue, allowing moderators to edit them and save or delete them

• New Option - Show Link Status Tooltip

• Improved handling of "429 Too Many Requests" status
• Fix link status being invalid on front-end

• Minor UI enhancements
• Support for servers not supporting HEAD requests

• New option: User Agent

• Bug-fix: search on Batch Update page triggered when all fields are empty

• Saving Batch Update form content between pages
• New option: Preview link limit
• Limit of items to process

• Bug-fix: Replacement code not stored in Action Log properly when deleting links from the Batch Update page

• Bug-fix: Data too long for column 'last_status_message'

• Bug-fix: only 40 links are deleted when batch processing the links

• Minor feature - Delete Action Log button

• Bug-fix: domain name longer than 255 characters
• Bug-fix: Mysqli statement execute error : Incorrect string value: '...' for column 'last_status_message' at row 1

• Dead-lock error fix
• Remove link text via Batch Update mode

• Added Skip Validation checkbox
• Removed Check Status button on Edit Link page. The status will be re-checked by default on save, unless Skip Validation option is used.
• Added an option to disable all front-end functions
• Added a checkbox on Batch Update page to filter the results for external links only

• Bugfix: link index is deleted in some cases

• Bug-fix: exclusion list not working

• Bug: Check Links does not update link status
• Bug: Deleting thread does not delete links from the database
• Bug: Links containing some special characters in the text are not detected

• New option: URL exclusion list

• 403 Forbidden status is considered as Valid
• User Agent is changed to AddonsLab/URLStatusChecker/1.0
• Bug-fix: Link statuses not updating after Check Links link is clicked

• Bug-fix: license shown as invalid in footer

• Action Log
• Improved support for unicode/special characters in URLs

• Bug-fix: slow query on loading the posts

• Bug-fix MySQL Error Data too long for column 'serializedlinks' at row 1

1. A new section in Admin Panel, under Tools tab, allowing you control the product.
2. Rebuild page to find all links posted on your board currently. A separate rebuild task can be run to check the status of the links.
3. Manage Links page lists all links found so far. It has a quick search and filter controls allowing you to find any particular links filtering by status or URL. Current link status is shown in the list, and broken links are highlighted for quicker orientation.
4. Each link can be edited individually. Edit page gives full information about last status check, allows to force-update the status by checking it one more time, to set the URL and text of the link or to delete it. It also has a link to the content where the URL was used so you can check the post quickly if needed.
5. Deletion of links allows setting a placeholder message that will replace the link. It can be left empty for the simple removal of the link.
6. Advanced batch update tool allows you to change an unlimited number of URLs at once. Caution: use the tool with care as it modifies the database content, and any submitted changes will not be reversible.
7. Batch update page allows to quickly find any link by its status code, the content of the URL or its text. Replacement options are to replace the URL/text entirely, or to do partial replacement only replacing the parts defined in your search criteria.
8. Regular expressions are supported in both search and replace mode. Subpatterns of matched searched criteria can be used as variables in the replacement.
9. A link can be shown under each post allowing to force-update the status of all links found in the post. Controllable by a new usergroup permission.
10. Links in the posts can have a "valid" or a "invalid" icon next to the link based on their status. The usergroups allowed to see the icons are controlled via XenForo Usergroup Permissions page.
11. The "valid" icon can be hidden for all usergroups at once if you prefer not to have it but to show broken links only.