Simple multi-line string manipulation from bash
I have a simple bash script for locating Raspberry Pis on the network: #!/bin/bash PIMAC=B8:27:EB sudo nmap -sP 192.168.251.0/24 | grep -i " $PIMAC" -A0 -B2 Whilst the output gives me the answer I want I'd like to tidy up the formatting. The current result is: Nmap scan report for 192.168.10.20 Host is up (-0.10s latency). MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation) -- Nmap scan report for 192.168.10.24 Host is up (-0.10s latency). MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation) What I want out of it is something more like: MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation) IP Address: 192.168.10.20 IP Address: 192.168.10.24 What is the best way to do this? Or should I just ditch the bash script for a Python script? -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
Or should I just ditch the bash script for a Python script?
Yes :-) If you want to use bash, you could parse the file, sort it, then present it like this: while read -r line; do if grep -q 'Nmap scan report for' <<<"$line"; then ip=$(sed 's/Nmap scan report for //' <<<"$line"); elif grep -q 'MAC Address' <<<"$line"; then mac=$(sed 's/MAC Address: //' <<<"$line") echo "$ip $mac" >> tmp1-$$ fi done sort -k 2 tmp1-$$ > tmp2-$$ lastmac='' while read -r line; do ip="$(sed 's/ .*//' <<<"$line")" mac="$(sed -E 's/^[^ ]+ //' <<<"$line")" if [ "$lastmac" != "$mac" ]; then echo "MAC Address: $mac" fi echo "IP Address: $ip" lastmac=$mac done <tmp2-$$ rm tmp1-$$ tmp2-$$ You could improve that by using bash 4 associate arrays rather than the intermediate files: #!/bin/bash declare -A arr while read -r line; do if grep -q 'Nmap scan report for' <<<"$line"; then ip=$(sed 's/Nmap scan report for //' <<<"$line"); elif grep -q 'MAC Address' <<<"$line"; then mac=$(sed 's/MAC Address: //' <<<"$line") arr["$mac"]="${arr["$mac"]} $ip" fi done for mac in "${!arr[@]}"; do printf "MAC Address: $mac\n" for ip in ${arr["$mac"]}; do printf "IP Address: $ip\n" done done If you prefer pipes to code, you could hack something like this: <input (grep -E '(MAC Address|Nmap scan report for)' | sed -E -e 's/MAC Address: (.*)/\1 END/' -e 's/Nmap scan report for (.*)/\1 /' | tr -d '\n') |sed 's/ END/\n/g' | sort -k 2 > a <a sed -E -e 's/^[^ ]+ //' | uniq | xargs -d '\n' -n 1 -I {} bash -c "echo 'MAC Address: {} '; grep '{}' a|sed -e 's/ .*//' -e 's/^/IP Address: /'" But it’s all pretty awful. I’m sure there somewhat better perl / awk ways to do it too. I’d run Nmap with `-oX n.xml` then use some python like: """ parse output of sudo nmap -sP 192.168.70.0/24 -oX - >n.xml """ import xml.etree.ElementTree as ET ROOT = ET.parse('n.xml').getroot() IPV4S = {} VENDORS = {} for h in ROOT.findall('host'): mac = None ipv4 = None vendor = 'unknown' for address in h.findall('address'): addrtype = address.attrib['addrtype'] if addrtype == "mac": mac = address.attrib['addr'] vendor = address.attrib['vendor'] elif addrtype == "ipv4": ipv4 = address.attrib['addr'] # print "mac={},ipv4={}".format(mac,ipv4) if mac in IPV4S: IPV4S[mac].append(ipv4) else: IPV4S[mac] = [ipv4] if mac not in VENDORS: VENDORS[mac] = vendor for mac in IPV4S: print "MAC Address: {} ({})".format(mac, VENDORS[mac]) print '\n'.join([" IP Address: {}".format(x) for x in IPV4S[mac]]) That way the parsing is more solid, and you can deal with edge cases better, and when you look at it in a few months it’s easier to see what’s going on. — Martijn
On 29 Jul 2019, at 13:44, Mark Rogers <mark@more-solutions.co.uk> wrote:
I have a simple bash script for locating Raspberry Pis on the network:
#!/bin/bash PIMAC=B8:27:EB sudo nmap -sP 192.168.251.0/24 | grep -i " $PIMAC" -A0 -B2
Whilst the output gives me the answer I want I'd like to tidy up the formatting. The current result is:
Nmap scan report for 192.168.10.20 Host is up (-0.10s latency). MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation) -- Nmap scan report for 192.168.10.24 Host is up (-0.10s latency). MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation)
What I want out of it is something more like: MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation) IP Address: 192.168.10.20 IP Address: 192.168.10.24
What is the best way to do this? Or should I just ditch the bash script for a Python script?
-- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
_______________________________________________ main@lists.alug.org.uk http://www.alug.org.uk/ https://lists.alug.org.uk/mailman/listinfo/main Unsubscribe? See message headers or the web site above!
On Mon, 29 Jul 2019 at 21:05, Martijn Koster <mak-alug@greenhills.co.uk> wrote:
Or should I just ditch the bash script for a Python script?
Yes :-)
Fair enough! Sorry for the delayed response, for some reason Gmail thought your reply was spam.
If you want to use bash, you could parse the file, sort it, then present it like this: [[..snip..]]
Thanks, I like that approach (in a "I used to push .bat files way beyond their comfort zone so this makes sense to me" kind of way), and there's some useful snippets in there I can re-use elsewhere, but the real message is that bash isn't really the best tool if I have Python installed anyway.
You could improve that by using bash 4 associate arrays rather than the intermediate files: [[..snip..]]
Now that's interesting, I haven't played with associative arrays in bash, and with that approach there's no compelling reason to use python to parse the human-readable nmap output. I'll definitely play with that, thanks!
If you prefer pipes to code, you could hack something like this: [[..snip..]]
One-liners are fun but these days I try to write scripts I have a hope of understanding once a few more grey cells have gone on their merry way!
I’d run Nmap with `-oX n.xml` then use some python like: [[..snip..]]
OK, so now there's a compelling reason to use python; XML does make more sense now you mention it. I think I'll work on this as my starting point. Thank you so much for the time and effort that went into that response. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
On Mon, Jul 29, 2019 at 01:44:51PM +0100, Mark Rogers wrote:
I have a simple bash script for locating Raspberry Pis on the network:
#!/bin/bash PIMAC=B8:27:EB sudo nmap -sP 192.168.251.0/24 | grep -i " $PIMAC" -A0 -B2
Whilst the output gives me the answer I want I'd like to tidy up the formatting. The current result is:
Nmap scan report for 192.168.10.20 Host is up (-0.10s latency). MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation) -- Nmap scan report for 192.168.10.24 Host is up (-0.10s latency). MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation)
What I want out of it is something more like: MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation) IP Address: 192.168.10.20 IP Address: 192.168.10.24
My script that uses nmap feeds the output into awk for processing, for straightforward selection and formmatting awk is sometimes the best approach. If more logic and decisions are involved then I move on to Python. My script just lists IP, host name, MAC and manufacturer. If not run as root then it just reports IP and host name. #!/bin/bash nmap -sP 192.168.1.0/24 | awk ' /Nmap scan report for/\ {if (NF == 6) printf("\n%-15.15s %-26.26s", substr(substr($6, 2), 1, length($6)-2), $5) else printf("\n%-43.43s", $5)} /MAC Address/\ {printf(" %s %s %s %s", $3, $4, $5, $6)} END {printf("\n")}' -- Chris Green
On Tue, 30 Jul 2019 at 09:30, Chris Green <cl@isbd.net> wrote:
My script just lists IP, host name, MAC and manufacturer. If not run as root then it just reports IP and host name.
#!/bin/bash [...]
Thanks Chris. I used to use awk a lot but it's a couple of decades ago and my memory is rusty, I think I need to get back into it. But my main challenge here is that I want to collate duplicates (so if one MAC is showing multiple IP addresses I want to list them together). Is this something that awk is suited to, or am I better of using python? -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
On Wed, Jul 31, 2019 at 12:31:16PM +0100, Mark Rogers wrote:
On Tue, 30 Jul 2019 at 09:30, Chris Green <cl@isbd.net> wrote:
My script just lists IP, host name, MAC and manufacturer. If not run as root then it just reports IP and host name.
#!/bin/bash [...]
Thanks Chris.
I used to use awk a lot but it's a couple of decades ago and my memory is rusty, I think I need to get back into it. But my main challenge here is that I want to collate duplicates (so if one MAC is showing multiple IP addresses I want to list them together). Is this something that awk is suited to, or am I better of using python?
Could you simply pipe the output into 'sort' with parameters to sort on the MAC field of your output? -- Chris Green
On Wed, 31 Jul 2019 at 13:19, Chris Green <cl@isbd.net> wrote:
Could you simply pipe the output into 'sort' with parameters to sort on the MAC field of your output?
Possibly, but Martijn has whet my appetite for trying an XML/Python approach now! :-) -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
On 29 Jul 13:44, Mark Rogers wrote:
I have a simple bash script for locating Raspberry Pis on the network:
#!/bin/bash PIMAC=B8:27:EB sudo nmap -sP 192.168.251.0/24 | grep -i " $PIMAC" -A0 -B2
Whilst the output gives me the answer I want I'd like to tidy up the formatting. The current result is:
Nmap scan report for 192.168.10.20 Host is up (-0.10s latency). MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation) -- Nmap scan report for 192.168.10.24 Host is up (-0.10s latency). MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation)
What I want out of it is something more like: MAC Address: B8:27:EB:22:1C:05 (Raspberry Pi Foundation) IP Address: 192.168.10.20 IP Address: 192.168.10.24
Well, personally, I'd do something like: --- BEGIN --- #!/bin/bash ip_mac_of_pi="$(sudo nmap -sP -n 192.168.1.0/24 | \ sed -ne '/Nmap scan report for / { s#Nmap scan report for ##; h; }; /MAC Address.*Raspberry Pi/ { s#MAC Address: ##; s# (.*$##; H; x; s#\n# #; p; }')" declare -A pi_macs while read -a ip_mac; do mac="${ip_mac[1]}" ip="${ip_mac[0]}" if [ -v "pi_macs[$mac]" ]; then pi_macs[$mac]+=" $ip" else pi_macs[$mac]="$ip" fi done <<< "$ip_mac_of_pi" for mac in "${!pi_macs[@]}"; do echo $mac: ${pi_macs[$mac]} done --- END --- Which will give them as mac: list of ips Thanks, -- Brett Parker
On Fri, 2 Aug 2019 at 14:07, Brett Parker <iDunno@sommitrealweird.co.uk> wrote:
Well, personally, I'd do something like: [[..snip..]]
That's a really concise example, thanks. I didn't know about bash arrays until I asked the initial question and I have a feeling I'll be using them quite a lot in future now. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0844 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
participants (4)
-
Brett Parker -
Chris Green -
Mark Rogers -
Martijn Koster