Archive and statistics » History » Version 17
Gašper Fele-Žorž, 15.02.2011 18:44
1 | 1 | Aleksander Bešir | h1. Archive and statistics |
---|---|---|---|
2 | 1 | Aleksander Bešir | |
3 | 2 | Aleksander Bešir | {{toc}} |
4 | 2 | Aleksander Bešir | |
5 | 5 | Janez Barbic | TBD - to be discussed |
6 | 5 | Janez Barbic | TODO - to do :) |
7 | 5 | Janez Barbic | Tole sem pustil zaenkrat not, da lahko hitro najdem stvari, ki jih moram se dopolnit oziroma jih moramo se predebatirat. V koncni verziji se bom teh kratic znebil in tudi tega komentarja v slovenscini :) |
8 | 1 | Aleksander Bešir | |
9 | 6 | Janez Barbic | h2. 1 Basic concept |
10 | 3 | Janez Barbic | |
11 | 11 | Janez Barbic | TODO Why syslog |
12 | 3 | Janez Barbic | |
13 | 11 | Janez Barbic | p. Eneraptor is designed in a way that it could communicate with any device regardless of its operating system. Syslog, system for logging program messages offers a quick and easy solutions for such communications, because it allows separation of the software that generates messages from the system that stores them and the software that reports and analyzes them. |
14 | 1 | Aleksander Bešir | |
15 | 11 | Janez Barbic | TODO malo nerodno zapisano? |
16 | 11 | Janez Barbic | |
17 | 3 | Janez Barbic | h3. 1.1 Hardware |
18 | 1 | Aleksander Bešir | |
19 | 11 | Janez Barbic | p. We will be using syslog-ng (syslog from now on), which has a few advantages over conventional syslog. Most basic setup , originator - collector, suits our needs just fine so far, but it can also be extended in the future. Syslog daemon (collector) will run on proxy server, syslog client (originator) will run on FRI-SMS system. |
20 | 1 | Aleksander Bešir | |
21 | 11 | Janez Barbic | TBD collector will be listening to a certain port (5140 - most commonly used) for incoming messages from client(s). |
22 | 11 | Janez Barbic | |
23 | 5 | Janez Barbic | Client mode operation: |
24 | 1 | Aleksander Bešir | |
25 | 5 | Janez Barbic | !https://lusy.fri.uni-lj.si/redmine/attachments/11/client_server_syslog.png! |
26 | 1 | Aleksander Bešir | |
27 | 1 | Aleksander Bešir | p. In client mode, syslog-ng collects the local logs generated by the host and forwards them through a network connection to the central syslog-ng server. Clients can also log the messages locally into files. |
28 | 5 | Janez Barbic | |
29 | 1 | Aleksander Bešir | |
30 | 11 | Janez Barbic | TODO more details on syslog-ng advantages over conventional syslog? |
31 | 5 | Janez Barbic | |
32 | 1 | Aleksander Bešir | h3. 1.2 Syslog message format |
33 | 11 | Janez Barbic | |
34 | 12 | Janez Barbic | TODO Mogoce tega sploh ne bo treba vedet, program() |
35 | 11 | Janez Barbic | |
36 | 5 | Janez Barbic | TBD TCP/IP vs UDP |
37 | 5 | Janez Barbic | TBD How do we plan to send messages to our daemon, via client (not always possible) or do we generate them ourselves? |
38 | 5 | Janez Barbic | TBD Using logfiles vs logging into database |
39 | 5 | Janez Barbic | |
40 | 5 | Janez Barbic | h3. 1.3 Development steps |
41 | 5 | Janez Barbic | |
42 | 5 | Janez Barbic | # Installing and configuring Syslog-ng daemon on proxy |
43 | 5 | Janez Barbic | # Installing and configuring Syslog-ng client on FRI-SMS |
44 | 5 | Janez Barbic | # TBD Designing Statistical Data Analysis Software to work with data collected by syslog daemon |
45 | 5 | Janez Barbic | # TBD Implementing SDA software |
46 | 1 | Aleksander Bešir | # TBD Testing? |
47 | 6 | Janez Barbic | |
48 | 6 | Janez Barbic | |
49 | 6 | Janez Barbic | h2. 2 Installing and configuring syslog-ng daemon on proxy |
50 | 7 | Janez Barbic | |
51 | 7 | Janez Barbic | p. Syslog-ng supports wide variety of Operating Systems so we aren't really restricted here. Since our proxy and FRI-SMS both run Linux 2.6 our obvious choce is syslog-ng for Linux. |
52 | 9 | Janez Barbic | |
53 | 1 | Aleksander Bešir | h3. 2.1 syslog-ng daemon on proxy |
54 | 9 | Janez Barbic | |
55 | 7 | Janez Barbic | Configuration file location: |
56 | 7 | Janez Barbic | /opt/syslog-ng/etc/syslog-ng.conf |
57 | 7 | Janez Barbic | |
58 | 8 | Janez Barbic | Basic syslog configuration: |
59 | 16 | Janez Barbic | |
60 | 7 | Janez Barbic | <pre><code class="ruby"> |
61 | 7 | Janez Barbic | @version: 3.2 |
62 | 7 | Janez Barbic | #Default configuration file for syslog-ng. |
63 | 7 | Janez Barbic | # |
64 | 7 | Janez Barbic | # For a description of syslog-ng configuration file directives, please read |
65 | 7 | Janez Barbic | # the syslog-ng Administrator's guide at: |
66 | 7 | Janez Barbic | # |
67 | 7 | Janez Barbic | # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html |
68 | 7 | Janez Barbic | # |
69 | 1 | Aleksander Bešir | |
70 | 1 | Aleksander Bešir | options { |
71 | 16 | Janez Barbic | time_reap(30); |
72 | 16 | Janez Barbic | mark_freq(10); |
73 | 16 | Janez Barbic | keep_hostname(yes); |
74 | 16 | Janez Barbic | }; |
75 | 1 | Aleksander Bešir | |
76 | 1 | Aleksander Bešir | ###### |
77 | 1 | Aleksander Bešir | # sources |
78 | 1 | Aleksander Bešir | source s_local { |
79 | 16 | Janez Barbic | # message generated by Syslog-NG |
80 | 16 | Janez Barbic | internal(); |
81 | 16 | Janez Barbic | # standard Linux log source (this is the default place for the syslog() |
82 | 16 | Janez Barbic | # function to send logs to) |
83 | 16 | Janez Barbic | unix-stream("/dev/log"); |
84 | 16 | Janez Barbic | # messages from the kernel |
85 | 16 | Janez Barbic | file("/proc/kmsg" program_override("kernel")); |
86 | 1 | Aleksander Bešir | }; |
87 | 1 | Aleksander Bešir | |
88 | 16 | Janez Barbic | # source s_syslog { syslog(ip(127.0.0.1) port(1999) transport("tcp")); }; |
89 | 1 | Aleksander Bešir | |
90 | 16 | Janez Barbic | ######################## |
91 | 16 | Janez Barbic | # Filters |
92 | 16 | Janez Barbic | ######################## |
93 | 16 | Janez Barbic | # Here's come the filter options. With this rules, we can set which |
94 | 16 | Janez Barbic | # message go where. |
95 | 16 | Janez Barbic | |
96 | 16 | Janez Barbic | # messages for eneraptor should have priority level LOG_MAIL and contain keyword "eneraptor" |
97 | 16 | Janez Barbic | filter f_mail_eneraptor { facility(mail) and match("eneraptor"); }; |
98 | 16 | Janez Barbic | |
99 | 9 | Janez Barbic | ###### |
100 | 9 | Janez Barbic | # destinations |
101 | 16 | Janez Barbic | destination d_messages { file("/var/log/messages"); }; |
102 | 16 | Janez Barbic | destination filtered_messages { file("/var/log/messages_filtered"); }; |
103 | 9 | Janez Barbic | |
104 | 9 | Janez Barbic | # |
105 | 16 | Janez Barbic | # SQL logging support |
106 | 9 | Janez Barbic | # |
107 | 9 | Janez Barbic | |
108 | 16 | Janez Barbic | destination d_pgsql { |
109 | 16 | Janez Barbic | sql(type(pgsql) |
110 | 16 | Janez Barbic | host("127.0.0.1") port(5432) username("eneraptor") password("eneraptor") |
111 | 16 | Janez Barbic | database("eneraptordb") |
112 | 16 | Janez Barbic | table("logs") |
113 | 16 | Janez Barbic | columns("datetime varchar(16)", "host varchar(32)", "program varchar(80)", "pid varchar(80)", "message varchar(200)") |
114 | 16 | Janez Barbic | values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSG") |
115 | 16 | Janez Barbic | indexes("datetime", "host", "program", "pid", "message")); |
116 | 9 | Janez Barbic | }; |
117 | 9 | Janez Barbic | |
118 | 16 | Janez Barbic | log { |
119 | 16 | Janez Barbic | source(s_local); |
120 | 16 | Janez Barbic | destination(d_messages); |
121 | 16 | Janez Barbic | # destination(d_pgsql); |
122 | 9 | Janez Barbic | }; |
123 | 9 | Janez Barbic | |
124 | 9 | Janez Barbic | log { |
125 | 16 | Janez Barbic | source(s_local); |
126 | 16 | Janez Barbic | filter(f_mail_eneraptor); |
127 | 16 | Janez Barbic | destination(filtered_messages); |
128 | 16 | Janez Barbic | destination(d_pgsql); |
129 | 9 | Janez Barbic | }; |
130 | 16 | Janez Barbic | |
131 | 9 | Janez Barbic | </code></pre> |
132 | 16 | Janez Barbic | |
133 | 16 | Janez Barbic | h3. 2.2 syslogd configuration on FRI-SMS |
134 | 16 | Janez Barbic | |
135 | 17 | Gašper Fele-Žorž | Edit <pre>/etc/inittab</pre> - change: |
136 | 17 | Gašper Fele-Žorž | <code><pre> |
137 | 17 | Gašper Fele-Žorž | null::respawn:/sbin/syslogd -n -m 0 |
138 | 17 | Gašper Fele-Žorž | </code></pre> |
139 | 1 | Aleksander Bešir | |
140 | 17 | Gašper Fele-Žorž | to |
141 | 17 | Gašper Fele-Žorž | |
142 | 17 | Gašper Fele-Žorž | <code><pre> |
143 | 17 | Gašper Fele-Žorž | null::respawn:/sbin/syslogd -n -m 0 -R 192.168.10.1 |
144 | 17 | Gašper Fele-Žorž | </code></pre> |
145 | 17 | Gašper Fele-Žorž | |
146 | 17 | Gašper Fele-Žorž | where <pre>192.168.10.1</pre> is the IP of the server running syslog-ng. Finally, |
147 | 17 | Gašper Fele-Žorž | have init reload it's configuration by running: |
148 | 17 | Gašper Fele-Žorž | |
149 | 17 | Gašper Fele-Žorž | <code><pre> |
150 | 17 | Gašper Fele-Žorž | kill -1 1 |
151 | 17 | Gašper Fele-Žorž | </code></pre> |
152 | 14 | Janez Barbic | |
153 | 13 | Janez Barbic | syslog-ng restart command: |
154 | 1 | Aleksander Bešir | |
155 | 1 | Aleksander Bešir | <pre><code>/etc/init.d/syslog-ng restart |
156 | 10 | Janez Barbic | </code></pre> |
157 | 10 | Janez Barbic | |
158 | 10 | Janez Barbic | h2. Statistical Data Analysis Software |
159 | 10 | Janez Barbic | |
160 | 11 | Janez Barbic | TODO |
161 | 11 | Janez Barbic | |
162 | 11 | Janez Barbic | h2. References |
163 | 1 | Aleksander Bešir | |
164 | 1 | Aleksander Bešir | "Syslog protocol, RFC 5424":http://tools.ietf.org/html/rfc5424 |
165 | 1 | Aleksander Bešir | "Syslog-ng homepage":http://www.balabit.com/network-security/syslog-ng |
166 | 1 | Aleksander Bešir | "Syslog-ng administrator guide":http://www.balabit.com/support/documentation/syslog-ng-ibm-agent-guide-admin-en.pdf |