Web Proxy Log –view Using mikrotik

Introduction

Since you probably use one or more of Mikrotik's routerboards and their RouterOS you,
as a network administrator, would probably be interested at one point what web sites your
network users visit. WebProxy Log can help you with that!

WebProxy Log package consist of WebProxy Log and WebProxy Log Catcher - intended
to receive, import, store Mikrotik's web-proxy log data and finally, generate reports from
those data. Both programs are unofficial (means not made by or supported by Mikrotik
company), freeware and made to use with Windows (the newer the better ;).

WebProxy Log Catcher

This module requires simple configuration and does the following: Listens on a certain IP
address/UDP port for log data which your pre-configured Mikrotik's logging environment
is sending. Once WebProxy Log Catcher 'catches' the data, it writes that data to a file on
your hard drive (to a log file).

WebProxy Log

WebProxy Log is a primary program which imports log files from the import location to
it's local database and can generate: surf logs (certain surfer's Internet traffic report
between two dates), top 10 visited servers pie chart, top 10 surfers pie chart and the 'Who
visited' report (which users visited certain server for how many times).

System Requirements:

• Windows XP, Vista or 7 (x86, x64)

• Few megs of RAM
• Whatever CPU
• Internet Explorer 6 or higher
• Adobe Flash Player

You can see screenshots and download the package to try it out yourself.

If you wish you can discuss about this software at our discussion site.

Check out Wiki Help if you need any.

Logging_environment
Learn to set up Logging environment
Deprecated
Updated Nov 5, 2010 by extensis
You'll need

• access to your Mikrotik

• WebProxy Log
• WebProxy Log Catcher

Mikrotik configuration

First of all, if you are not using manual client-side proxy configuration, you'll need to set
up transparent proxy on your Mikrotik. You can learn here more about it.

When you're sure that clients will make traffic through proxy, you'll have to set up
logging in your Mikrotik. Either by using the console or WinBox:

• go to 'System/Logging/Actions'
• add new Action with following details:

• Name: WebProxyLog
• Type: remote
• Remote Address: enter here the IP address of the remote computer where
WebProxy Log Catcher will be running
• Remote Port: enter here UDP port of the computer where WebProxy Log
Catcher will be listening from the remote computer - usually for syslog 514

• now go to 'System/Logging/Rules'
• add new rule with following details:
 • Topics: web-proxy, !debug
• Prefix: proxy
• Action: WebProxyLog (or whatever name you gave to the action)

Make sure that UDP port you specified for remote logging is open both in
Mikrotik's firewall and in remote computer's firewall!

WebProxy Log Catcher configuration

• download and install WebProxy Log package

• when you run WebProxy Log Catcher for the first time, the settings dialog will
pop up (or you can select 'Cofigure' option by right-clicking WebProxy Log
Catcher's icon in system tray)
 • configure IP address, UDP port and logs locations (where IP addres and UDP port
are located on compter where WebProxy Log Catcher is running - the same IP
address and UDP port you specified for the remote computer in you Mikrotik)
• create folder a folder name Logs as the location show in the picture. Then
automatically create and folder name Daily and Logs between logs file .You will
show effect after 24 Hours.( hanif’s noet)
• once you apply configuration, you are ready to start!

That's it!

IP_Alias
Learn to configure IP aliases list
Deprecated
Updated Jun 25, 2010 by extensis

You'll need

• WebProxy Log

About

Since you don't want in your report to see IP addresses as users, you can open IP alias
configuration (menu 'Settings/IP aliases') and attach there an alias to a certain IP address.
Reports will then show names instead of IP addresses.

IP addresses are automatically added to this list when importing a log file, or manually
added with the '+' button. Removing an alias is easy; select a field containing an IP
address or alias and click '-' button.

You can export IP aliases list to a CSV file by clicking 'Export', and import the same
CSV file by clicking 'Import'.

'Frob DB' button searches your database for missing IP addresses that are not in the list
and adds if any is found.

Whether or not you make any changes, if you don't click 'Apply' button, when closing the
IP aliases dialog, no changes will be made. Logs
Learn to work with logs
Deprecated
Updated Jun 26, 2010 by extensis

You'll need

• configured Logging environment

• log data for at least one day (download zipped sample log)
 • WebProxy Log

Importing logs

• open WebProxy Log

• go to 'File\Import logs'
• click Import

All log files (any file that has 'log' extension) from pre-configured import location will
then be read by WebProxy Log, reformatted and stored in local database. Imported log
files, since they're stored in database, can be deleted or archived (recommended).

Deleting records

• open WebProxy Log

• go to 'File\Delete records'
• select whether to delete all records for certain user
• select whether to delete certain web page records
• once you've selected what to delete, press 'Delete' button

When you press the 'Delete' button, you are prompted to confirm deletion of n records
found in the database.

When deleting records, you can backup database (this is done BEFORE actual deletion)
and to optimize database after deletion.

Optimization is a process which cleans your database of unneeded records (the deleted
ones) and which defragments the database (can take while on large databases).

You'll need

• WebProxy Log
• imported logs

Once you have imported logs and attached IP aliases, you are ready to generate some
report.

Internet traffic report

This report shows the list of visited websites for selected user within selected dates.

To generate this report, first select user from a user drop-down list in the main window,
select from and to dates (this can be done by clicking-and-dragging on the calendar),
select some filters (if you want), and click 'Generate' button.

Once the log is displayed, you can:

 • Open a web page directly from the report (by clicking a server name)
• Use right-click on a server name and select 'Copy shortcut', once shortcut is
copied, you can add it to the filter list by clicking '+' button
• Print the report
• Save the report

Print and Save is available for 'Internet traffic' and 'Who visited...' reports only!

Who visited... report

This report shows a list of which users visited a certain page for how many times.

To generate this report go to 'Reports/Who visited...', in the dialog enter what page are
you interested for (eg. '*google.com') and click OK.

Top 10 reports

This reports show a pie-charts that represent how many times each server has been visited
or which user has visited most pages.

For 'Top 10 servers' report you can use Site filter (eg. filter out unwanted servers in your
report).

To generate these reports, go to 'Reports/Top 10 servers' or 'Reports/Top 10 surfers'.

Backup database

'Backup database' option can be activated at some points when you're uncertain of some
actions (eg. when deleting records). When 'Backup database' option is selected, a copy of
your database file will be made in WebProxy Log's configuration directory.

Optimize

Optimization is a process which cleans your database of unneeded records (the deleted
ones) and which defragments the database (can take a while on large databases).

Insecure import

'Insecure import' is actually not so insecure and can't lead to database corruption. It's only
a different approach when writing data to the database.

When you have this option turned off, WebProxy Log will wait for database to finish
writing each line of data and then send new line of data to be written. That is not such a
big deal if you have small log files (up to 2 MB daily), but on large log files (above 5 MB
daily) the import process takes a log time to finish.
Therefore you can use 'insecure import' so WebProxy Log will send all the lines of data
to the database without waiting each of them to be actually written. This 'insecure import'
drastically speeds up the import process.