Tuesday 1 December 2020

Filtering and Collecting the Results from the Network Scanner

 by Vaggelis Atlasis 

Previously, I added a function that sends a TCP packet to every IP address in the subnet in an attempt to initiate hand-checking. Using a dictionary with common TCP ports, the packet was sent to each and every one of the ports. If there was a response, it was classified as as one of types of response: the host listening in the port, the host rejection connection in the port or the host prohibiting communication on the port. 

The next step, was filtering the useful from the two packets (ping, TCP) we send to each address in the subnet and save them in afile. 

Setting Up the Collection of Our Results

The information we are mainly interested from each subnet is the addresses that responded, their hostnames, their reachability and the TCP ports we receive replies from. I decide that the best way to collect the information was through a dictionary, making the IP address the key and the rest of the information the value, as a list. That way it would be easy to assign the gathered information to the corresponding IP address without worrying about each address being assigned pieces of information separately, or replacing the existing information like it could in a list. 

Therefore, in the function send_ping, I intialised the dictionary results, where all the results would be collected. The reason results was intialised inside the function was that this was the function were we obtained all the details for each address (including the TCP related information). Next, we create the key for each address in the subnet with the line results[str(address)] = "", making the address the key of the dictionary and leaving its value empty for now. We turn it into a string for convenience and to avoid potential errors. 

Formatting the Information Obtained from the TCP Function

In order to make the results from the send_tcp function available to store in a variable in the send_ping function, we need to return them. However, as there may be multiple ports that respond for a given address, it is efficient and convenient to store them in a list, which we call tcp_open_ports. Hence, every time we get a response with a TCP layer from the host, we run the line tcp_open_ports.append(p) where p is the port that responded from the TCP common ports list. At the end of the function, the list is returned with the line return tcp_open ports

Collecting the Results

With the return of TCP ports now in place, we can now run the function whenever a host reponds to the ping packet and is reachable. In that case, we store the returned information from the send_TCP function to a variable named tcp_open_ports. With the line results[str(address))] = [host_name, "Reachable", tcp_open_ports], we make the value of the given address all of the wanted information in the form of a list. The host name and reachability (given by the send_ping function) and the TCP ports (given by the send_tcp function). 

In the case that communication is prohibited by the host, we set the value of the given address as results[str(address))] = [host_name, "ICMP Uncreachable"]. There is no need to run the send_tcp function as we will receive no reply. We also specify that it was the ICMP protocol (part of the ping packet) that is unreachable to avoid confusion. 

Finally, if there is no response by the address, the value of the IP address is simply given as results[str(address))] = [host_name, "No response"]

The dictionary results with all the collected information is then returned by the function. 

The updated version of the two functions mentioned above. New bits are placed inside the red boxes (click to enlarge. Note: there should a tcp_open_ports.append(p) in all of the selection loops in the send_tcp function - except the last one). 



Saving the Results in a Text File

The easiest way to save our results in a text file is using the csv (comma-separated value) library. Therefore, at the top of the program we import csv and later on define the function saveResults, with results as a parameter, of course. 

Next we use the lines you see in the picture below to open the file in order to write to it using the csv writer. We start by writing the row Ip Address, Host Name, Reachability, TCP Open Ports as subtitles for the file. Then, I used a nested for loop; the first one to write the keys and their respective values in a different line and the second one to count the length of the value of each key, which is a list. If the length of the list is 3, we just write each item in the list separately (for formatting pouproses). If not, the length will be 2. In that case, an empty pair of square brackets is printed to indicate that there was no TCP response. 

The code for the saveResults function which saves the results in a csv file. 



Finalising Everything 

As results is being returned from the send_ping function, it needs to be stored in a variable so that we can access it. Therefore, I went to where the function is called (process.map(send_ping, targets) and simply declared it as a variable. That way, the returned information can be stored and accessed at will. 

Finally, with a few lines of code almost identical to the saveResults function above, the results are printed in a user-friendly display. 

The new code in the main function (top line in not new but I added it for reference).


The Code 

from scapy.all import IP, ICMP, TCP, sr1, UDP, DNS, RandShort, DNSQR
from ipaddress import IPv4Network, ip_address
from multiprocessing import Pool
from argparse import ArgumentParser
import csv

dns_server=None

tcp_common_ports = {
    21: "ftp",
    22: "ssh",
    23: "telnet",
    25: "smtp",
    53: "domain name system",
    80: "http",
    110: "pop3",
    111: "rpcbind",
    135: "msrpc",
    139: "netbios-ssn",
    143: "imap",
    443: "https",
    445: "microsoft-ds",
    993: "imaps",
    995: "pop3s",
    1723: "pptp",
    3306: "mysql",
    3389: "ms-wbt-server",
    5900: "vnc",
    8080: "http-proxy" }

#the following function will to resolve an IPv4 address to its host name
def resolve_IP(ip_address_to_resolve):
    global dns_server
    reverse_ip_address = ip_address(ip_address_to_resolve).reverse_pointer
    #print("The reverse pointer of" , ip_address_to_resolve , "is" , reverse_ip_address)
    packet = IP(dst = dns_server)/UDP(dport = 53, sport = RandShort())/DNS(rd = 1, qd = DNSQR(qname = reverse_ip_address, qtype = "PTR"))
    response = sr1(packet, verbose = 0, timeout = 2)
    host_name = "not resolved"
    if response:
        if int(response.getlayer(UDP).sport) == 53:
            try:
                #print("The host name of" , ip_address_to_resolve , "is" , response.an.rdata.decode('utf-8'))
                host_name = response.an.rdata.decode('utf-8')
            except Exception as e:
                pass
    return host_name

       
def send_ping(address):
    results = {}
    results[str(address)]=[]
    packet = IP(dst = str(address))/ICMP(type = 8, code = 0)
    response = sr1(packet, verbose = 0, timeout = 2)
    host_name = resolve_IP(address)
    if response:
        if int(response.getlayer(ICMP).type) == 0:
            print("Host" , address , "is reachable. Its host name is" , host_name)
            tcp_open_ports = send_tcp(address)
            results[str(address)]=[host_name,"Reachable", tcp_open_ports]
        elif int(response.getlayer(ICMP).type) == 3:
            print("Destination" , address , "is unreachable")
            results[str(address)]=[host_name,"ICMP Unreachable"]
    else:
        results[str(address)]=[host_name,"No response"]
    return results
 
   
def send_tcp(address):
    ports = list(tcp_common_ports.keys()) ##returns a list of all keys (aka port numbers) of tcp_common_ports dictionary
    tcp_open_ports = []
    for p in ports:
        packet = IP(dst = str(address))/TCP(sport = RandShort(),dport = p, flags = "S")
        response = sr1(packet, verbose = 0, timeout = 2)
        if response:
            if response.haslayer(TCP):
                if str(response.getlayer(TCP).flags) == "SA":
                    print("Host ", address , "is listening in port",p,tcp_common_ports[p])
                    tcp_open_ports.append(p)
                elif str(response.getlayer(TCP.flags)) == "R":
                    print("Host ", address , "is rejecting conenction on port",p,tcp_common_ports[p])
            else:
                if int(response.getlayer(ICMP).type) == 3 and int(response.getlayer(ICMP).code) == 13:
                    print("Host " , address, ": communication prohibited on port",p,tcp_common_ports[p])
                else:
                    print(response.summary())
    return tcp_open_ports

def saveResults(results):
    with open ("scanner_results.csv", "w") as file:
        write = csv.writer(file)
        write.writerow(["IP Address", "Host Name", "Reachability",  "TCP Open ports"])
        for r in results:
            for k in r:
                if len(r[k]) == 3:
                    write.writerow([k, r[k][0], r[k][1], r[k][2]])
                else:
                    write.writerow([k, r[k][0], r[k][1], "[]"])
       

def main():
    parser = ArgumentParser()
    parser.add_argument("-p", "--processes" , help = "the number of processes to run in parallel (must be integer)"  , type = int, default = 10)
    parser.add_argument("-dns", "--dns_server" , help = "the DNS server to be used for our querries"  , type = str, default = "8.8.8.8")
    values = parser.parse_args()
   
    global dns_server
    dns_server = values.dns_server
   
    network = input("Please enter the IPv4 network you'd like to scan (e.g. 192.168.1.0/24) or just an IPv4 address: ")
    valid_target = False
    while valid_target == False:
        try:
            targets = IPv4Network(network)
            valid_target = True
        except Exception as e:
            print(e)
            network = input("Please enter a valid IPv4 subnet: ")
           
    with Pool(values.processes) as process:
        results = process.map(send_ping, targets)
   
    print()
    print("IP Address \t Host Name \t Reachability \t TCP Open ports")
    print("---------------------------------------------------------------------")
    for r in results:
        for k in r:
            if len(r[k]) == 3:
                print(k, "\t", r[k][0],"\t", r[k][1], "\t", r[k][2])
            else:
                print(k, "\t", r[k][0],"\t", r[k][1], "\t", "[]")
    saveResults(results)

if __name__ == "__main__":
    main()

For Next-Time

By the next blog post, I will have uploaded the code to GitHub, where it will be available and accessible more easily. I will also look to improve and develop the code further. 

Saturday 21 November 2020

Participation at ESERO CanSat Competition

BSN competes at CanSat competition

by Vaggelis Atlasis

Together with our schoolmates at the British School of the Netherlands Antoine and Ayrton and under the mentoring of our teacher Mr Hurley we submitted a proposal for participating at the Dutch CanSat competition, forming the team of "the Canservationists". A few days ago we were happy to hear the news that our proposal has been accepted for the next phase of the competition. 

CanSat is an ESERO project executed by NEMO on the instructions of the Netherlands SpaceOffice (NSO). The Ministry of Defense is the logistic partner during the test day and launch event. The aim of the competition is to spark young people’s enthusiasm for a career in technology. 

The competition is a simulation of a real satellite. The challenge of this competition is to fit all the major subsystems of a satellite inside the volume and shape of a soft drink can, which will be launched by a small rocket up to an altitude of 1 km. CanSat is equipped with an Arduino and various sensors.  

Each team has to accomplish two missions: the first, common for all of the teams, is to collect air temperature and pressure data as the CanSat descends. The second is chosen by the teams themselves. Our team will collect carbon dioxide and potentially other greenhouse  gases so as to correlate them with the temperature and the atmospheric pressure. Our objective is to better understand the global warming effect. 

In addition, as part of our secondary mission we will implement a tele-command system that will be operated via radio control. This system will allow us to change the rate at which we take measurements from our CanSat sensors. We will also integrate a Navigation sensor so that during the flight of the CanSat we may map our recorded data. Using this sensor, we will also be able to track our CanSat and find its location during its flight and after it lands.

(picture from https://www.esa.int/Education/CanSat_overview)

Our mission has been inspired by the ESA Copernicus Sentinel-5P satellite, which maps Air Quality ratings over a large part of Europe and surrounding areas. 

Currently we are in the design phase, choosing our sensors, finding sponsors to fund our project, designing the parachute, etc. 

We will keep you posted ;-) 

Filippos & Vaggelis Atlasis

members of “the  Canservationists”

Sunday 20 September 2020

Turning the Ping Scanner into a Network Scanner

by Vaggelis Atlasis

Previously I added some lines of code so that the ping scanner could utilise a DNS server and reverse look-up the hostnames of the responsive IP address. Up until that point, the scanner only utilised the ICMP protocol to send packets. As a new addition to the program, a TCP packet was sent to the selected addresses in the subnet along with the ICMP one, initiating a communication process between the two computers known as handshaking.

TCP 3-Way Handshaking

Handshaking is the process that occurs before two computers that use the TCP protocol start communicating. It involves Computer A, sending an SYN or S flag (TCP uses flag for the same purposes ICMP uses types) to Computer B. This lets B know that A wants to communicate with B. In the case that B also wants to communicate with A, he sends a packet with two flags; an ACK or A flag - used to let A know that B has acknowledged the message - as well as an SYN flag, communicating to computer A, returning the favour (this is an ACK/SYN packet) Finally, computer A sends an ACK packet back to B and communication between the two computers begins. 

Note that handshaking only takes place in TCP and not in UDP as UDP prioritises fast delivery over making sure all packets arrive.

A simple diagram showing how handshaking works in TCP



Finding the Most Common Scanned Ports 

We begin by importing TCP  from the scapy. all module. In order to make the process a little more specific, I looked for the most popular ports when it came to network scanning. I found, what I believed was, an appropriate list containing 20 ports (for a list of all TCP and UDP ports take a look at this Wikipedia page) and created a dictionary with the port number as the key and the protocol it corresponded to as the value. 

A dictionary named "tcp_common_ports" containing the 20 TCP ports



Sending the TCP Packet

Firstly, we define the function send_tcp with address as its parameter where n IP address will be inputted. Next, a list is initialised using the dictionary keys (which are the port numbers). 

For every port, p, in the list, a packet is sent to the designated IP address fro ma random source port to the destination port p. As we want to initiate the handshaking process, the flag in the packet is "S". We use sr1 to send the packet made above and wait for a response. The waiting time per packet is 2 seconds as indicated by the timeout in the variable. We name the whole process response. If we get an answer from the host, the response is true. 

If it is true, the program looks at whether the response contains a TCP layer. If that is the case, the program then looks for a TCP layer with the "SA"  - SYN and ACK - flag, indicating that computer B has acknowledged the packet we sent and has accepted the handshake. We then output a message that this certain IP address is listening on port p (number), which corresponds to a certain protocol in the tcp_common_ports dictionary. 

In the case, that there is no packet with the "SA" flag, we look for an "R" flag instead which would indicate the host is rejecting the connection. Just like above, we display am message wit the address of the host and the port the packet was sent to (the number and the corresponding protocol); only this time we let the user know that he host is rejecting connection on that port. 

Moreover, we often found that the host responded with an ICMP layer of type 3 and code 13 (destination unreachable, communication prohibited). Therefore, if the response did not contain a TCP layer, we checked for a response containing the layer mentioned above. If that was the case, a message stating that the host of that IP address prohibited communication on port p

Finally, if none of the cases mentioned above was true, the program prints a summary of the response so that it can be added to the possible list of replies. 
The function for sending TCP packets

The function is then implemented in the send_ping function that already existed under the line where an ICMP type 0 response (Echo-reply) is received. Therefore, the function is only called if the host replies to an ICMP packet Echo-request packet with an Echo-reply one.

Current Version of the Program 

As the program has now become too big to fit into one picture, I will leave a copy-pasted version of it. 

from scapy.all import IP, ICMP, TCP, sr1, UDP, DNS, RandShort, DNSQR
from ipaddress import IPv4Network, ip_address
from multiprocessing import Pool
from argparse import ArgumentParser

dns_server=None

tcp_common_ports = {
    21: "ftp",
    22: "ssh",
    23: "telnet",
    25: "smtp",
    53: "domain name system",
    80: "http",
    110: "pop3",
    111: "rpcbind",
    135: "msrpc",
    139: "netbios-ssn",
    143: "imap",
    443: "https",
    445: "microsoft-ds",
    993: "imaps",
    995: "pop3s",
    1723: "pptp",
    3306: "mysql",
    3389: "ms-wbt-server",
    5900: "vnc",
    8080: "http-proxy" } 

#the following function will to resolve an IPv4 address to its host name
def resolve_IP(ip_address_to_resolve):
    global dns_server 
    reverse_ip_address = ip_address(ip_address_to_resolve).reverse_pointer
    #print("The reverse pointer of" , ip_address_to_resolve , "is" , reverse_ip_address)
    packet = IP(dst = dns_server)/UDP(dport = 53, sport = RandShort())/DNS(rd = 1, qd = DNSQR(qname = reverse_ip_address, qtype = "PTR"))
    response = sr1(packet, verbose = 0, timeout = 2)
    host_name = "not resolved"
    if response:
        if int(response.getlayer(UDP).sport) == 53:
            try:
                #print("The host name of" , ip_address_to_resolve , "is" , response.an.rdata.decode('utf-8'))
                host_name = response.an.rdata.decode('utf-8')
            except Exception as e:
                pass
    return host_name

        
def send_ping(address):
    packet = IP(dst = str(address))/ICMP(type = 8, code = 0)
    response = sr1(packet, verbose = 0, timeout = 2)
    if response:
        if int(response.getlayer(ICMP).type) == 0:
            host_name = resolve_IP(address)
            print("Host" , address , "is reachable. Its host name is" , host_name)
            send_tcp(address)
        elif int(response.getlayer(ICMP).type) == 3:
            print("Destination" , address , "is unreachable")

def send_tcp(address):
    ports = list(tcp_common_ports.keys()) ##returns a list of all keys (aka port numbers) of tcp_common_ports dictionary
    for p in ports:
        packet = IP(dst = str(address))/TCP(sport = RandShort(),dport = p, flags = "S")
        response = sr1(packet, verbose = 0, timeout = 2)
        if response:
            if response.haslayer(TCP):
                if str(response.getlayer(TCP).flags) == "SA":
                    print("Host ", address , "is listening in port",p,tcp_common_ports[p])
                elif str(response.getlayer(TCP.flags)) == "R":
                    print("Host ", address , "is rejecting conenction on port",p,tcp_common_ports[p])
            else:
                if int(response.getlayer(ICMP).type) == 3 and int(response.getlayer(ICMP).code) == 13:
                    print("Host " , address, ": communication prohibited on port",p,tcp_common_ports[p])
                else:
                    print(response.summary())
        

def main():
    parser = ArgumentParser()
    parser.add_argument("-p", "--processes" , help = "the number of processes to run in parallel (must be integer)"  , type = int, default = 10)
    parser.add_argument("-dns", "--dns_server" , help = "the DNS server to be used for our querries"  , type = str, default = "8.8.8.8")
    values = parser.parse_args()
   
    global dns_server 
    dns_server = values.dns_server

    network = input("Please enter the IPv4 network you'd like to scan (e.g. 192.168.1.0/24) or just an IPv4 address: ")
    valid_target = False
    while valid_target == False:
        try:
            targets = IPv4Network(network)
            valid_target = True
        except Exception as e:
            print(e)
            network = input("Please enter a valid IPv4 subnet: ")
            
    with Pool(values.processes) as process:
        process.map(send_ping, targets)


if __name__ == "__main__":
    main()




Wind turbine experiment


by Filippos and Vaggelis Atlasis 

We conducted an experiment to find out which wind turbine design generates the most energy. We used the EV3 energy pack to build the turbine and a fan to create "wind". Attached below you will find our full report created as a pdf as well as some pictures and videos of our experiment. The report contains everything from an introduction and method, to our analysis and a conclusion.









Friday 14 August 2020

Ping Scanner Part 3: Using a DNS Server to resolve Responsive IP Addresses

 by Vaggelis Atlasis


Previously, I built on the ping scanner by adding the code necessary for multi-processing, making the ping-sweep process faster. The next step was to implement the code for a reverse look-up, using the responsive IP addresses to resolve their hostnames.

The previous version of the code

Resolving the IP Address

To be able to access and utilise a DNS server, I imported DNS, DNSQR, UDP and RandShort from Scapy, as well as ip_address from ipaddress. I began by initialising the variable dns_server to enable the user to choose a DNS server of their choice later on in the program. 

I then defined the function resolve_IP with ip_address_to_resolve as a parameter in order to resolve a given ip_address. I globalised the variable dns_server as the program got confused and printed an error in line 13 (dst = dns_server). For the IP address to be resolved, it first needed to be reversed. I used the ip_address().reverse_pointer function to take in an IP address and return the reverse version of the function (e.g. 192.168.1.0 becomes 0.1.168.192.in-addr.arpa).

I then initialise the packet which is to be sent as a query to the server. First, for the IP protocol (line 13), we set the destination to the one of the server chosen by the user. For UDP - which is the transfer protocol DNS servers use - I set the destination port to 53, which is the one DNS has been designed to use with UDP and TCP. Finally, I use RandShort to choose a source port at random. Finally, we need to assign the details for the DNS protocol, however, we will explore them in another post later on. In line 15, I set the verbose (amount of unnecessary/excess information) to 0 and the timeout (how much to wait for a response) to 2 seconds. Next, I set the default of the hostname of the IP address to "not resolved" (why will be explained soon). 

If there is a response by the DNS server, the program checks that the response is an integer UDP response with a source port of 53. Then, the program tries to replace the previous hostname with the one returned by the DNS server. If this is not possible, the exception is passed. The hostname is then returned to the program. 

Implementing a DNS request using a function (Lines 13 and 14 should be one combined line, they were only split so that the full line would fit into the picture).

Implementing the Resolving Function in the Ping Function

The ping function remained largely unchanged. The only difference is that in line 33, with address as the value for ip_address_to_resolve. The next change is not necessary to the program but makes it more user friendly as the hostname is displayed after a responsive IP address. If there was no hostname returned by the DNS server earlier, the default message ("not resolved" is printed instead.

The updated ping function

Allowing the User to Input a DNS Server from the Command Line

Using an argument parser, the user can use -dns or --dns_server to input a string of characters corresponding to a DNS server of their choice from the command line. Should they choose not to input anything, the default value for the server is "8.8.8.8" (Google's DNS server). This argument is then assigned to the variable values. Finally, dns_server is globalised and assigned the value of values.dns_server (the .dns_server part is necessary so that only the DNS server is assigned). From there, when the function is run, the DNS server the user has inputted (or the default server) will be the DNS server used in the program. 

Parsing an argument from the command line and assigning it to the value for the dns_server (line 41 and 42 should be one line but were split so that all of the code fit in the snapshot)

Up Next 

In the next post, we will TCP scan for the open ports for each of the hosts we've identified. I will also provide an explanation on the details of the DNS protocol when it comes to packets.