I mentioned above that I’m learning regular expressions, so I had a look at Marc Anthony’s script for that reason. It correctly returns the IP addresses, but it appears to return every other number that contains 4 or more digits.
I rewrote the regular expression, and it seemed to work as desired. It was also the fastest of the working suggestions.
do shell script "echo " & my (read (choose file))'s quoted form & " | egrep -o '(\\d{1,3}\\.){3}\\d{1,3}' | sort -h | uniq >> " & ((path to desktop as text) & "IP.txt")'s POSIX path's quoted form
If you search for ‘regex ip address’ or stuff like that, you’ll find many pages that discuss this. While the regex used here works generally, it won’t exclude incorrect addresses (e.g. those with number sets > 255) but that doesn’t look like it would be an issue here and it is less complex than those that do incorporate that constraint.
I originally posted a correctly working regex, edited it to shorten the pattern, realized the error after supplemental testing, and changed it back shortly after. You captured the code in flux.
That’s a little surprising, as TIDs tend to be very speedy, but… woohoo! Chalk one one up for shell scripting.
I’m submitting the IP’s which are attacking and it is working well. However I’m told that some of them are duplicates of those already on the master list of IP’s. So would it be possible for the script to access another list of IPs and remove any that are already on that list?
Thanks in advance for any thoughts. Oh if it helps the preferred script I’m using is this one (as it avoids conflicts with Bitdefender):
use framework "Foundation"
use scripting additions
on open theDroppedItems
set theFile to POSIX path of item 1 of theDroppedItems
set theIPFile to getFileName(theFile)
set theText to (current application's NSString's stringWithContentsOfFile:theFile encoding:(current application's NSUTF8StringEncoding) |error|:(missing value))
set theIPData to getIPData(theText)
(current application's NSString's stringWithString:theIPData)'s writeToFile:theIPFile atomically:true encoding:(current application's NSUTF8StringEncoding) |error|:(missing value)
end open
on getIPData(theText)
set regExPattern to "[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+"
set theRegEx to current application's NSRegularExpression's regularExpressionWithPattern:regExPattern options:0 |error|:(missing value)
set regExMatches to theRegEx's matchesInString:theText options:0 range:{location:0, |length|:theText's |length|()}
set ipList to {}
repeat with anItem in regExMatches
set end of ipList to (theText's substringWithRange:(anItem's range())) as text
end repeat
set ipSet to current application's NSOrderedSet's orderedSetWithArray:ipList
set ipSortedArray to ipSet's array()'s sortedArrayUsingSelector:"localizedStandardCompare:"
return ((ipSortedArray's componentsJoinedByString:linefeed) as text)
end getIPData
on getFileName(theFile)
set theFile to current application's NSString's stringWithString:theFile
set fileBase to theFile's stringByDeletingPathExtension()
set fileExtension to theFile's pathExtension()
return ((fileBase's stringByAppendingString:"_IP_CLEANED")'s stringByAppendingPathExtension:fileExtension)
end getFileName
To do this, you’d better write the filtering results of each text not to some new file, but to the existing properties list (.plist) of your script (application). This will change the script itself a lot. There are enough users here who can help with this. I have no time now.
Other choice is: use one persistent text file for filtering results (that is, the name and location hard-coded). When dropping the unfiltered text file on your droplet, it should firstly read contents of persistent text file into the oldIPs list variable, then append only new non-duplicate IPs to oldIPs, then sort, then write the updated list back to the persistent text file. This will change the script itself a lot as well. I have no time now.
I think I misread what the second option was. The master (deny) IP list is not maintained by myself but a third party. That list is updated weekly (if not daily) with all the IPs that need to be blocked on a Synology (or similar) NAS.
What I’m now looking to do is to download the latest incarnation of that list and then process my IP list from my Orbi router, remove the duplicates and sort it (as we do now) and then open the third party deny list and remove any IP addresses which are already in the master deny list.
That way any IPs I submit are only those that are new and need including in the master list. This way the author of the master deny list does not have to check all my submissions manually where a great number may be duplicated. As an example last week I submitted 25 IPs for blocking and all 25 it turns out were already on the list.
dbrewood. I’ve modified my script to work as you want. The script prompts for the IP deny list, which seems a bit awkward, but the actual path can be placed in the script instead. The IP deny list has to be paragraph-separated text of IP addresses only. I tested the script and it worked without issue.
use framework "Foundation"
use scripting additions
on open theDroppedItems
set theFile to POSIX path of item 1 of theDroppedItems
set theExistingFile to POSIX path of (choose file)
set theIPFile to getFileName(theFile)
set theText to (current application's NSString's stringWithContentsOfFile:theFile encoding:(current application's NSUTF8StringEncoding) |error|:(missing value))
set theExistingText to (current application's NSString's stringWithContentsOfFile:theExistingFile encoding:(current application's NSUTF8StringEncoding) |error|:(missing value))
set theIPData to getIPData(theText, theExistingText)
(current application's NSString's stringWithString:theIPData)'s writeToFile:theIPFile atomically:true encoding:(current application's NSUTF8StringEncoding) |error|:(missing value)
end open
on getIPData(theText, theExistingText)
set regExPattern to "[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+"
set theRegEx to current application's NSRegularExpression's regularExpressionWithPattern:regExPattern options:0 |error|:(missing value)
set regExMatches to theRegEx's matchesInString:theText options:0 range:{location:0, |length|:theText's |length|()}
set ipSet to current application's NSMutableSet's new()
repeat with anItem in regExMatches
(ipSet's addObject:(theText's substringWithRange:(anItem's range())))
end repeat
set ipExistingArray to (theExistingText's componentsSeparatedByCharactersInSet:(current application's NSCharacterSet's newlineCharacterSet()))
set ipExistingSet to current application's NSSet's setWithArray:ipExistingArray
ipSet's minusSet:ipExistingSet
set ipSortedArray to ipSet's allObjects()'s sortedArrayUsingSelector:"localizedStandardCompare:"
return ((ipSortedArray's componentsJoinedByString:linefeed) as text)
end getIPData
on getFileName(theFile)
set theFile to current application's NSString's stringWithString:theFile
set fileBase to theFile's stringByDeletingPathExtension()
set fileExtension to theFile's pathExtension()
return ((fileBase's stringByAppendingString:"_IP_CLEANED")'s stringByAppendingPathExtension:fileExtension)
end getFileName
Wow! That is totally awesome and does indeed look to do what is needed. I’ll not know for sure until I run it with real data, but on test data I just threw together it looked to work perfectly!
If I do want to hard code the path for the third party deny list:
use framework "Foundation"
use scripting additions
on open theDroppedItems
set theFile to POSIX path of item 1 of theDroppedItems
# set theExistingFile to POSIX path of (choose file)
set theExistingFile to "/Users/dbrewood/Library/Mobile Documents/com~apple~CloudDocs/_Daron Files/NAS/Deny List/deny-ip-list.txt"
set theIPFile to getFileName(theFile)
set theText to (current application's NSString's stringWithContentsOfFile:theFile encoding:(current application's NSUTF8StringEncoding) |error|:(missing value))
set theExistingText to (current application's NSString's stringWithContentsOfFile:theExistingFile encoding:(current application's NSUTF8StringEncoding) |error|:(missing value))
set theIPData to getIPData(theText, theExistingText)
(current application's NSString's stringWithString:theIPData)'s writeToFile:theIPFile atomically:true encoding:(current application's NSUTF8StringEncoding) |error|:(missing value)
end open
on getIPData(theText, theExistingText)
set regExPattern to "[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+"
set theRegEx to current application's NSRegularExpression's regularExpressionWithPattern:regExPattern options:0 |error|:(missing value)
set regExMatches to theRegEx's matchesInString:theText options:0 range:{location:0, |length|:theText's |length|()}
set ipList to {}
repeat with anItem in regExMatches
set end of ipList to (theText's substringWithRange:(anItem's range())) as text
end repeat
set ipArray to current application's NSMutableArray's arrayWithArray:ipList
set ipExistingText to current application's NSString's stringWithString:theExistingText
set newlineSet to current application's class "NSCharacterSet"'s newlineCharacterSet()
set ipExistingArray to (ipExistingText's componentsSeparatedByCharactersInSet:(newlineSet))
ipArray's removeObjectsInArray:ipExistingArray
set ipSet to current application's NSOrderedSet's orderedSetWithArray:ipArray
set ipSortedArray to ipSet's array()'s sortedArrayUsingSelector:"localizedStandardCompare:"
return ((ipSortedArray's componentsJoinedByString:linefeed) as text)
end getIPData
on getFileName(theFile)
set theFile to current application's NSString's stringWithString:theFile
set fileBase to theFile's stringByDeletingPathExtension()
set fileExtension to theFile's pathExtension()
return ((fileBase's stringByAppendingString:"_IP_CLEANED")'s stringByAppendingPathExtension:fileExtension)
end getFileName
Which works very well indeed. However I’m looking to automate the process further by submitting the list of addresses to www.abuseipdb.com via their API (detailed here: https://docs.abuseipdb.com/#configuring-fail2ban). What I’d want to be getting back from each IP address is something like this:
185.162.235.175 was found in our database, this IP was reported 110 times. Confidence of Abuse is 100% - ISP Alliance LLC - Country Netherlands - City Meppel, Drenthe
This would make reporting to the guys who compile the IP ban list very easy.
I am not familiar with the site’s API, and I would not like to study it, and then also pay (if the key is paid). Therefore, I solved this problem with simple parsing (which can be improved further). But if there is an expert on this API, then his solution will probably be better than mine.
set ipAddress to "95.217.31.46"
tell application "Safari"
open location "https://www.abuseipdb.com/check/" & ipAddress
activate
my waitSafariWebPageLoading()
set HTML to text of document 1
close document 1
end tell
set ATID to AppleScript's text item delimiters
set AppleScript's text item delimiters to {"CHECK
" & ipAddress, "REPORT " & ipAddress & "
WHOIS " & ipAddress}
set theReport to text item 2 of HTML
set AppleScript's text item delimiters to ATID
activate me
display dialog ipAddress & " " & theReport buttons {" Cancel ", " OK "}
on waitSafariWebPageLoading()
tell application "System Events" to tell application process "Safari"
repeat until ((UI element "Reload this page" of group 3 of toolbar 1 of window 1 exists) or (UI element "Reload this page" of group 2 of toolbar 1 of window 1 exists))
delay 0.1
end repeat
end tell
end waitSafariWebPageLoading
Weird, I’ve tried the revised script on macOS Monterey 12.3.1 and it stops at the same point which is where the web page loads and shows the reported IP address.
I have to force quit the script at that point to get it to exit.
Okay I’ve downloaded the script and made it readable. I’ve got the Apple script as:
set ipListFile to POSIX path of (choose file of type ".csv")
set resultsFile to (POSIX path of (path to desktop folder)) & "results.csv"
do shell script "/Users/myusername/Library/Mobile Documents/com~apple~ScriptEditor2/Documents/BulkAbuseCheck.sh" & quoted form of ipListFile & " " & quoted form of resultsFile
When I run it and get the file selector up and I look to choose the file all files are greyed out and not selectable.
The source I have is a .txt file, and I then imported that into a spreadsheet and exported it out as a .csv file. Same problem, both files are greyed out and can’t be selected?