In my inputs.conf file, if I have "no_appending_timestamp = true" as shown in the documentation, no graphs are created. The data is parsed fine, just no graphs.
When I go to say, the System Dashboard and do an "Open in Search" I see the search starts with " | tstats count (log_subtype) as ce from pan_system". If I try doing just that portion, the result is zero. Yet I can do a search on "sourcetype = pan_system" and see multiple values for log_subtype.
I looked more at the tstats function and saw that it depends on (time series) tsindex files. I looked in /opt/splunk/var/lib/splunktsidxstatspan_system and saw no tsindex files.
I decided to modify inputs.conf, commenting out "no_appending_timestamp = true". When I restarted the app, then I saw that tsindex files were being created and I was getting graphs. However the receive_time field was being populated not with the value in the original syslog message, but with value that was prepended by "no_appending_timestamp".
So I don't understand - the documentation says to have "no_appending_timestamp = true", but that produces no graphs (in my case) and if I decide to comment it out, then at least one of the fields are not parsed correctly.
Here is an example syslog message with "no_appending_timestamp = true":
<11>Mar 10 10:58:50 192.168.62.3 1,2014/03/10 10:58:50,000FD103199,SYSTEM,general,0,2014/03/10 10:58:50,,general,,0,0,general,high,"Failed to connect to Pan-Agent at 192.168.62.4, source: 192.168.62.3 (41 times)",0,0x0
And here is the same with "no_appending_timestamp = true" commented out:
Mar 5 21:19:09 192.168.62.5 <11>Mar 5 21:19:09 192.168.62.3 1,2014/03/05 21:19:09,0004C102557,SYSTEM,general,0,2014/03/05 21:19:09,,general,,0,0,general,high,"Failed to connect to Pan-Agent at 192.168.62.4, source: 192.168.62.3 (41 times)",0,0x0
Any help would be appreciated.