synchronising videos with dataDumper logs

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

synchronising videos with dataDumper logs

Konstantinos Theofilis
Hi list,

I have recorded logs from the robot in an experiment using dataDumper. At the same time, a video was taken of the experiment using a normal video camera.

I have a system that lets me know the yarp timestamp of one specific frame of the video acquired. E.g. I know that frame x_1 (not the first frame of the actual video file though) of the video file has a yarp timestamp of t_1. Subsequently, I know the timestamp t_n of the other frames x_n, by adding (1000/FPS)*(n-1) milliseconds to t_1.

What I want to do is to be able to replay the video in a synchronised fashion with dataSetPlayer playing the logs from the robot.

I suppose that one way is to use yarpdev ffpmeg to convert the video to a series of YarpImageOf bottles and record them with dataDumper. Then (with a script) modify the timestamps accordingly. But in that case, 
as the video is 1080p, I am worried that any missing frames in the output of the driver will distort the timestamps. Videos are approximately 6-7 mins long.

Another way would be to convert the video file to ppm images, then adding the timestamps, since I know the timestamp t_1 of frame x_1, in a dataDumper compatible format.

My issue with both the above approaches is the amount of space needed, since I have over 100 videos.

Is there a reasonably accurate way to achieve what I want, without resorting to the "video->ppm->manually (script) add timestamps" process?

Best regards,
Kostas Theofilis


------------------------------------------------------------------------------

_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: synchronising videos with dataDumper logs

Lars Schillingmann
Hi Konstantinos and all,

A while ago I also thought about that, and it seems indeed an issue for most video processing tools to process external presentation time stamps (PTS), etc.. The datadumper usually writes a file data.log and a file video.avi (HUFFYUV encoding, http://wiki.icub.org/iCub_documentation/group__dataDumper.html) if it is run with the corresponding options to save video data. The logfile contains the timestamps. Since you are anyway able to acquire the timestamps, I just mentioned it to be complete.

mkvmerge is able to process external timestamp information. I use the following python snippet to create a file that creates a compatible timestamp file from a datadumper data.log file:
print '# timecode format v2'
prevts = 0.0
for x in file('data.log'):
        ts = float(x.split()[1])
        if prevts == 0.0:
                prevts = ts
        print "%0.0f" % ((ts - prevts)*1000)

The timecode format is documented here:
https://www.bunkus.org/videotools/mkvtoolnix/doc/mkvmerge.html
(see 18.2)

Assuming that ts.list contains the timestamps in the format described above, mkvmerge can be called like this:
mkvmerge -o out.mkv --timecodes 0:ts.list video.avi

After that the mkv file contains the correct timing information for each frame. Since many tools cannot handle variable frame rates it should be processed with ffmpeg. ffmpeg's default behavior duplicates frames where necessary to achieve a constant frame rate. Some compression e.g. h264 also makes sense in most cases.
E.g. ffmpeg -i out.mkv -crf 18 -preset ultrafast out.mp4

Another approach could be to use gstreamer but I wanted to rely on simpler tools only. A slight disadvantage of the above approach is the temporary mkv file.

Best
Lars


On 2014-08-16 00:34 +0900, Konstantinos Theofilis wrote:

> Hi list,
>
> I have recorded logs from the robot in an experiment using dataDumper. At the same time, a video was taken of the experiment using a normal video camera.
>
> I have a system that lets me know the yarp timestamp of one specific frame of the video acquired. E.g. I know that frame x_1 (not the first frame of the actual video file though) of the video file has a yarp timestamp of t_1. Subsequently, I know the timestamp t_n of the other frames x_n, by adding (1000/FPS)*(n-1) milliseconds to t_1.
>
> What I want to do is to be able to replay the video in a synchronised fashion with dataSetPlayer playing the logs from the robot.
>
> I suppose that one way is to use yarpdev ffpmeg to convert the video to a series of YarpImageOf bottles and record them with dataDumper. Then (with a script) modify the timestamps accordingly. But in that case,  as the video is 1080p, I am worried that any missing frames in the output of the driver will distort the timestamps. Videos are approximately 6-7 mins long.
>
> Another way would be to convert the video file to ppm images, then adding the timestamps, since I know the timestamp t_1 of frame x_1, in a dataDumper compatible format.
>
> My issue with both the above approaches is the amount of space needed, since I have over 100 videos.
>
> Is there a reasonably accurate way to achieve what I want, without resorting to the "video->ppm->manually (script) add timestamps" process?
>
> Best regards,
> Kostas Theofilis
>
>
>
> ------------------------------------------------------------------------------
>
>
>
> _______________________________________________
> Robotcub-hackers mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
>

 

------------------------------------------------------------------------------
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: synchronising videos with dataDumper logs

Konstantinos Theofilis
Hi Lars,

thanks for your comprehensive answer.

So by following your suggestion, I can produce a video file that incorporates the yarp timestamps (just a reminder, we are talking about the original video acquired by a normal video camera, not running through some YARP grabber).

However, it is not clear to me how i can play that video in a synchronised fashion with the logged data from dataDumper (without converting it beforehand to separate images).

E.g. is it possible to have dataSetPlayer to play the video (with the incorporated timestamps, or the timestamps in an external data.log file)?

Best regards,
Kostas Theofilis

P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file? Like: 1) Converting ppm images to an avi 2) Converting data.log to ts.list 3) mkvmerge the avi with ts.list 4) Final video with corrected timing info



On 16/08/14 16:17, Lars Schillingmann wrote:
Hi Konstantinos and all,

A while ago I also thought about that, and it seems indeed an issue for most video processing tools to process external presentation time stamps (PTS), etc.. The datadumper usually writes a file data.log and a file video.avi (HUFFYUV encoding, http://wiki.icub.org/iCub_documentation/group__dataDumper.html) if it is run with the corresponding options to save video data. The logfile contains the timestamps. Since you are anyway able to acquire the timestamps, I just mentioned it to be complete.

mkvmerge is able to process external timestamp information. I use the following python snippet to create a file that creates a compatible timestamp file from a datadumper data.log file:
print '# timecode format v2'
prevts = 0.0
for x in file('data.log'):
        ts = float(x.split()[1])
        if prevts == 0.0:
                prevts = ts
        print "%0.0f" % ((ts - prevts)*1000)

The timecode format is documented here:
https://www.bunkus.org/videotools/mkvtoolnix/doc/mkvmerge.html
(see 18.2)

Assuming that ts.list contains the timestamps in the format described above, mkvmerge can be called like this:
mkvmerge -o out.mkv --timecodes 0:ts.list video.avi

After that the mkv file contains the correct timing information for each frame. Since many tools cannot handle variable frame rates it should be processed with ffmpeg. ffmpeg's default behavior duplicates frames where necessary to achieve a constant frame rate. Some compression e.g. h264 also makes sense in most cases.
E.g. ffmpeg -i out.mkv -crf 18 -preset ultrafast out.mp4

Another approach could be to use gstreamer but I wanted to rely on simpler tools only. A slight disadvantage of the above approach is the temporary mkv file.

Best
Lars


On 2014-08-16 00:34 +0900, Konstantinos Theofilis wrote:
Hi list,

I have recorded logs from the robot in an experiment using dataDumper. At the same time, a video was taken of the experiment using a normal video camera.

I have a system that lets me know the yarp timestamp of one specific frame of the video acquired. E.g. I know that frame x_1 (not the first frame of the actual video file though) of the video file has a yarp timestamp of t_1. Subsequently, I know the timestamp t_n of the other frames x_n, by adding (1000/FPS)*(n-1) milliseconds to t_1.

What I want to do is to be able to replay the video in a synchronised fashion with dataSetPlayer playing the logs from the robot.

I suppose that one way is to use yarpdev ffpmeg to convert the video to a series of YarpImageOf bottles and record them with dataDumper. Then (with a script) modify the timestamps accordingly. But in that case,  as the video is 1080p, I am worried that any missing frames in the output of the driver will distort the timestamps. Videos are approximately 6-7 mins long.

Another way would be to convert the video file to ppm images, then adding the timestamps, since I know the timestamp t_1 of frame x_1, in a dataDumper compatible format.

My issue with both the above approaches is the amount of space needed, since I have over 100 videos.

Is there a reasonably accurate way to achieve what I want, without resorting to the "video->ppm->manually (script) add timestamps" process?

Best regards,
Kostas Theofilis



------------------------------------------------------------------------------



_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers

 

------------------------------------------------------------------------------
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers


------------------------------------------------------------------------------

_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: synchronising videos with dataDumper logs

Lars Schillingmann
Hi Konstantinos,

On 2014-08-17 04:15 +0900, Konstantinos Theofilis wrote:
> So by following your suggestion, I can produce a video file that incorporates the yarp timestamps (just a reminder, we are talking about the original video acquired by a normal video camera, not running through some YARP grabber).

Hmm, ok it seems I did not get some important points of your mail. The idea of my approach was to convert the yarp recorded video data so it can be synchronized with the external camera. After missing frames and variable timing is compensated with this approach, it is easy to synchronize it, but it does not solve your problem...

> However, it is not clear to me how i can play that video in a synchronised fashion with the logged data from dataDumper (without converting it beforehand to separate images).

I did not try that, but I would neither send the data over yarp (because I would just cause load on the system sending around HD uncompressed images) nor use conversion to ppm. The first I would try is to patch the dataSetPlayer to include an external clock signal. Then I would use mplayer or mpv player etc. to play the HD video, read status information e.g. via status messages and use this information to trigger the dataSetPlayer to step forward. This way the external camera's video would be used as a kind of master clock for the replay process. If the video does not contain audio that should be played other custom solutions, e.g. using opencv to play the external video could be easy too.
If you want to somehow process the HD video data with another yarp based module, of course this solution does not make any sense. Extending the dataSetPlayer to read a video file might be necessary then.

> P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file? Like: 1) Converting ppm images to an avi 2) Converting data.log to ts.list 3) mkvmerge the avi with ts.list 4) Final video with corrected timing info

Yes that should work. The container for step 1 does not matter much, mkv would also do for step one.

Best
Lars

------------------------------------------------------------------------------
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: synchronising videos with dataDumper logs

Konstantinos Theofilis
Hi,

I went though the process to enable dataDumper to save videos (building ffmpeg and opencv from source). DataSetPlayer does not indeed replay videos, only images.

So, to recap and clarify:

1) For videos acquired by dataDumper or a yarp device, a data.log with timestamps have to be produced (if it does not exist) and then merged with the actual video file, through mkvmerge, in order to "normalise" the framerate. (While that was not my original question, it is very helpful though :) ).

2) For videos recorded with a camera normally (no connection to yarp, just writing directly to e.g., the sd card of the camera), a way to associate frames with yarp timestamps have to be used. Then, as my original thought before posting here and as you also mention, dataSetPlayer has to be patched to read the timestamp info from the video and syncronise the playing of the logged data with an mplayer instance (generally speaking).

By the way, my method for providing "externally acquired" videos with yarp timestamps, in case I want to get annotated data from these videos and associate this data with yarp-logged data is the following:

I have an arduino, with a pushbutton and a LED. The arduino is connected to a machine in the icub cluster through USB. The arduino is constantly sending to its serial port 0s, while the button is not pushed. When the button is pushed, the LED lights up and the board sends 1s.

At the same time, on the machine that the arduino is connected, a simple yarp python module is running that listens to the serial port (USB). This module get the input from the serial port. When it detects a 1, it plays a sound, sends a 1 to a port, e.g. /serial  and then quits after a few seconds.
The /serial port is also logged with dataDumper, along with all other inputs from the robot.

The arduino board is then used as a clapper board at the start of the experiment, putting it in the middle of the scene and pressing the button. So I get a visual indication (LED), an audio indication (the sound played when the button is pushed) and a log from dataDumper with a timestamp of when the "1" was received.

That way, I can find the frame of the video that the LED is lit and the audio is first heard and associate it with a yarp timestamp.

The above system was devised out of necessity, to be able to provide a bit more accurate timing info for annotated data, in a yarp timestamp format, so it is of more interest to people that also extract annotated data from external videos.

Anyway, I will try to write something to syncronise dataSetPlayer and mplayer. If I am succesful, I will submit a patch.

Thanks!
Kostas Theofilis




On 17/08/14 14:35, Lars Schillingmann wrote:
Hi Konstantinos,

On 2014-08-17 04:15 +0900, Konstantinos Theofilis wrote:
So by following your suggestion, I can produce a video file that incorporates the yarp timestamps (just a reminder, we are talking about the original video acquired by a normal video camera, not running through some YARP grabber).
Hmm, ok it seems I did not get some important points of your mail. The idea of my approach was to convert the yarp recorded video data so it can be synchronized with the external camera. After missing frames and variable timing is compensated with this approach, it is easy to synchronize it, but it does not solve your problem...

However, it is not clear to me how i can play that video in a synchronised fashion with the logged data from dataDumper (without converting it beforehand to separate images).
I did not try that, but I would neither send the data over yarp (because I would just cause load on the system sending around HD uncompressed images) nor use conversion to ppm. The first I would try is to patch the dataSetPlayer to include an external clock signal. Then I would use mplayer or mpv player etc. to play the HD video, read status information e.g. via status messages and use this information to trigger the dataSetPlayer to step forward. This way the external camera's video would be used as a kind of master clock for the replay process. If the video does not contain audio that should be played other custom solutions, e.g. using opencv to play the external video could be easy too.
If you want to somehow process the HD video data with another yarp based module, of course this solution does not make any sense. Extending the dataSetPlayer to read a video file might be necessary then.

P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file? Like: 1) Converting ppm images to an avi 2) Converting data.log to ts.list 3) mkvmerge the avi with ts.list 4) Final video with corrected timing info
Yes that should work. The container for step 1 does not matter much, mkv would also do for step one.

Best
Lars

------------------------------------------------------------------------------
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers


------------------------------------------------------------------------------

_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

Lorenzo Natale-2
In reply to this post by Konstantinos Theofilis

Dear Lars, Konstantinos,

Interesting discussion indeed.

 

> P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file?

 

The datadumper could dump directly an avi instead of individual frames. If somebody is willing to work on it, it would be very much appreciated.

 

Lorenzo

 

 

From: Konstantinos Theofilis [mailto:[hidden email]]
Sent: sabato 16 agosto 2014 21:16
To: [hidden email]
Subject: [LIKELY_SPAM]Re: [rc-hackers] synchronising videos with dataDumper logs

 

Hi Lars,

thanks for your comprehensive answer.

So by following your suggestion, I can produce a video file that incorporates the yarp timestamps (just a reminder, we are talking about the original video acquired by a normal video camera, not running through some YARP grabber).

However, it is not clear to me how i can play that video in a synchronised fashion with the logged data from dataDumper (without converting it beforehand to separate images).

E.g. is it possible to have dataSetPlayer to play the video (with the incorporated timestamps, or the timestamps in an external data.log file)?

Best regards,
Kostas Theofilis

P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file? Like: 1) Converting ppm images to an avi 2) Converting data.log to ts.list 3) mkvmerge the avi with ts.list 4) Final video with corrected timing info


On 16/08/14 16:17, Lars Schillingmann wrote:

Hi Konstantinos and all,
 
A while ago I also thought about that, and it seems indeed an issue for most video processing tools to process external presentation time stamps (PTS), etc.. The datadumper usually writes a file data.log and a file video.avi (HUFFYUV encoding, http://wiki.icub.org/iCub_documentation/group__dataDumper.html) if it is run with the corresponding options to save video data. The logfile contains the timestamps. Since you are anyway able to acquire the timestamps, I just mentioned it to be complete.
 
mkvmerge is able to process external timestamp information. I use the following python snippet to create a file that creates a compatible timestamp file from a datadumper data.log file:
print '# timecode format v2'
prevts = 0.0
for x in file('data.log'):
        ts = float(x.split()[1])
        if prevts == 0.0:
                prevts = ts
        print "%0.0f" % ((ts - prevts)*1000)
 
The timecode format is documented here:
https://www.bunkus.org/videotools/mkvtoolnix/doc/mkvmerge.html
(see 18.2)
 
Assuming that ts.list contains the timestamps in the format described above, mkvmerge can be called like this:
mkvmerge -o out.mkv --timecodes 0:ts.list video.avi
 
After that the mkv file contains the correct timing information for each frame. Since many tools cannot handle variable frame rates it should be processed with ffmpeg. ffmpeg's default behavior duplicates frames where necessary to achieve a constant frame rate. Some compression e.g. h264 also makes sense in most cases.
E.g. ffmpeg -i out.mkv -crf 18 -preset ultrafast out.mp4
 
Another approach could be to use gstreamer but I wanted to rely on simpler tools only. A slight disadvantage of the above approach is the temporary mkv file.
 
Best
Lars
 
 
On 2014-08-16 00:34 +0900, Konstantinos Theofilis wrote:
Hi list,
 
I have recorded logs from the robot in an experiment using dataDumper. At the same time, a video was taken of the experiment using a normal video camera.
 
I have a system that lets me know the yarp timestamp of one specific frame of the video acquired. E.g. I know that frame x_1 (not the first frame of the actual video file though) of the video file has a yarp timestamp of t_1. Subsequently, I know the timestamp t_n of the other frames x_n, by adding (1000/FPS)*(n-1) milliseconds to t_1.
 
What I want to do is to be able to replay the video in a synchronised fashion with dataSetPlayer playing the logs from the robot.
 
I suppose that one way is to use yarpdev ffpmeg to convert the video to a series of YarpImageOf bottles and record them with dataDumper. Then (with a script) modify the timestamps accordingly. But in that case,  as the video is 1080p, I am worried that any missing frames in the output of the driver will distort the timestamps. Videos are approximately 6-7 mins long.
 
Another way would be to convert the video file to ppm images, then adding the timestamps, since I know the timestamp t_1 of frame x_1, in a dataDumper compatible format.
 
My issue with both the above approaches is the amount of space needed, since I have over 100 videos.
 
Is there a reasonably accurate way to achieve what I want, without resorting to the "video->ppm->manually (script) add timestamps" process?
 
Best regards,
Kostas Theofilis
 
 
 
------------------------------------------------------------------------------
 
 
 
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
 
 
 
 
------------------------------------------------------------------------------
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers

 


------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

pattacini

Ciao Lorenzo,

 

Maybe I did not catch the point, but dataDumper is already able to produce an avi file with the options “--type video” or “--type image --addVideo”; the latter to produce both images and an avi.

 

Ciao,

Ugo

 

From: Lorenzo Natale [mailto:[hidden email]]
Sent: 29 August 2014 10:18
To: Konstantinos Theofilis; [hidden email]
Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

 

Dear Lars, Konstantinos,

Interesting discussion indeed.

 

> P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file?

 

The datadumper could dump directly an avi instead of individual frames. If somebody is willing to work on it, it would be very much appreciated.

 

Lorenzo

 

 

From: Konstantinos Theofilis [[hidden email]]
Sent: sabato 16 agosto 2014 21:16
To: [hidden email]
Subject: [LIKELY_SPAM]Re: [rc-hackers] synchronising videos with dataDumper logs

 

Hi Lars,

thanks for your comprehensive answer.

So by following your suggestion, I can produce a video file that incorporates the yarp timestamps (just a reminder, we are talking about the original video acquired by a normal video camera, not running through some YARP grabber).

However, it is not clear to me how i can play that video in a synchronised fashion with the logged data from dataDumper (without converting it beforehand to separate images).

E.g. is it possible to have dataSetPlayer to play the video (with the incorporated timestamps, or the timestamps in an external data.log file)?

Best regards,
Kostas Theofilis

P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file? Like: 1) Converting ppm images to an avi 2) Converting data.log to ts.list 3) mkvmerge the avi with ts.list 4) Final video with corrected timing info

On 16/08/14 16:17, Lars Schillingmann wrote:

Hi Konstantinos and all,
 
A while ago I also thought about that, and it seems indeed an issue for most video processing tools to process external presentation time stamps (PTS), etc.. The datadumper usually writes a file data.log and a file video.avi (HUFFYUV encoding, http://wiki.icub.org/iCub_documentation/group__dataDumper.html) if it is run with the corresponding options to save video data. The logfile contains the timestamps. Since you are anyway able to acquire the timestamps, I just mentioned it to be complete.
 
mkvmerge is able to process external timestamp information. I use the following python snippet to create a file that creates a compatible timestamp file from a datadumper data.log file:
print '# timecode format v2'
prevts = 0.0
for x in file('data.log'):
        ts = float(x.split()[1])
        if prevts == 0.0:
                prevts = ts
        print "%0.0f" % ((ts - prevts)*1000)
 
The timecode format is documented here:
https://www.bunkus.org/videotools/mkvtoolnix/doc/mkvmerge.html
(see 18.2)
 
Assuming that ts.list contains the timestamps in the format described above, mkvmerge can be called like this:
mkvmerge -o out.mkv --timecodes 0:ts.list video.avi
 
After that the mkv file contains the correct timing information for each frame. Since many tools cannot handle variable frame rates it should be processed with ffmpeg. ffmpeg's default behavior duplicates frames where necessary to achieve a constant frame rate. Some compression e.g. h264 also makes sense in most cases.
E.g. ffmpeg -i out.mkv -crf 18 -preset ultrafast out.mp4
 
Another approach could be to use gstreamer but I wanted to rely on simpler tools only. A slight disadvantage of the above approach is the temporary mkv file.
 
Best
Lars
 
 
On 2014-08-16 00:34 +0900, Konstantinos Theofilis wrote:
Hi list,
 
I have recorded logs from the robot in an experiment using dataDumper. At the same time, a video was taken of the experiment using a normal video camera.
 
I have a system that lets me know the yarp timestamp of one specific frame of the video acquired. E.g. I know that frame x_1 (not the first frame of the actual video file though) of the video file has a yarp timestamp of t_1. Subsequently, I know the timestamp t_n of the other frames x_n, by adding (1000/FPS)*(n-1) milliseconds to t_1.
 
What I want to do is to be able to replay the video in a synchronised fashion with dataSetPlayer playing the logs from the robot.
 
I suppose that one way is to use yarpdev ffpmeg to convert the video to a series of YarpImageOf bottles and record them with dataDumper. Then (with a script) modify the timestamps accordingly. But in that case,  as the video is 1080p, I am worried that any missing frames in the output of the driver will distort the timestamps. Videos are approximately 6-7 mins long.
 
Another way would be to convert the video file to ppm images, then adding the timestamps, since I know the timestamp t_1 of frame x_1, in a dataDumper compatible format.
 
My issue with both the above approaches is the amount of space needed, since I have over 100 videos.
 
Is there a reasonably accurate way to achieve what I want, without resorting to the "video->ppm->manually (script) add timestamps" process?
 
Best regards,
Kostas Theofilis
 
 
 
------------------------------------------------------------------------------
 
 
 
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
 
 
 
 
------------------------------------------------------------------------------
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers

 


------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

Konstantinos Theofilis
Hi all,

indeed, as Ugo says, dataDumper is already capable of producing an avi file. The only minor annoyance is that it needs (at least on debian/ubuntu systems) for ffmpeg and openCV to be compiled from source, which has more to do with the libav/ffmpeg situation in debian repositories and not dataDumper itself.

On the other hand, dataSet player cannot replay an avi file. That (for me at least) would be useful and I am in the process of implementing it.

Best,
Kostas Theofilis

On 29/08/14 10:23, Ugo Pattacini wrote:

Ciao Lorenzo,

 

Maybe I did not catch the point, but dataDumper is already able to produce an avi file with the options “--type video” or “--type image --addVideo”; the latter to produce both images and an avi.

 

Ciao,

Ugo

 

From: Lorenzo Natale [[hidden email]]
Sent: 29 August 2014 10:18
To: Konstantinos Theofilis; [hidden email]
Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

 

Dear Lars, Konstantinos,

Interesting discussion indeed.

 

> P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file?

 

The datadumper could dump directly an avi instead of individual frames. If somebody is willing to work on it, it would be very much appreciated.

 

Lorenzo

 

 

From: Konstantinos Theofilis [[hidden email]]
Sent: sabato 16 agosto 2014 21:16
To: [hidden email]
Subject: [LIKELY_SPAM]Re: [rc-hackers] synchronising videos with dataDumper logs

 

Hi Lars,

thanks for your comprehensive answer.

So by following your suggestion, I can produce a video file that incorporates the yarp timestamps (just a reminder, we are talking about the original video acquired by a normal video camera, not running through some YARP grabber).

However, it is not clear to me how i can play that video in a synchronised fashion with the logged data from dataDumper (without converting it beforehand to separate images).

E.g. is it possible to have dataSetPlayer to play the video (with the incorporated timestamps, or the timestamps in an external data.log file)?

Best regards,
Kostas Theofilis

P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file? Like: 1) Converting ppm images to an avi 2) Converting data.log to ts.list 3) mkvmerge the avi with ts.list 4) Final video with corrected timing info

On 16/08/14 16:17, Lars Schillingmann wrote:

Hi Konstantinos and all,
 
A while ago I also thought about that, and it seems indeed an issue for most video processing tools to process external presentation time stamps (PTS), etc.. The datadumper usually writes a file data.log and a file video.avi (HUFFYUV encoding, http://wiki.icub.org/iCub_documentation/group__dataDumper.html) if it is run with the corresponding options to save video data. The logfile contains the timestamps. Since you are anyway able to acquire the timestamps, I just mentioned it to be complete.
 
mkvmerge is able to process external timestamp information. I use the following python snippet to create a file that creates a compatible timestamp file from a datadumper data.log file:
print '# timecode format v2'
prevts = 0.0
for x in file('data.log'):
        ts = float(x.split()[1])
        if prevts == 0.0:
                prevts = ts
        print "%0.0f" % ((ts - prevts)*1000)
 
The timecode format is documented here:
https://www.bunkus.org/videotools/mkvtoolnix/doc/mkvmerge.html
(see 18.2)
 
Assuming that ts.list contains the timestamps in the format described above, mkvmerge can be called like this:
mkvmerge -o out.mkv --timecodes 0:ts.list video.avi
 
After that the mkv file contains the correct timing information for each frame. Since many tools cannot handle variable frame rates it should be processed with ffmpeg. ffmpeg's default behavior duplicates frames where necessary to achieve a constant frame rate. Some compression e.g. h264 also makes sense in most cases.
E.g. ffmpeg -i out.mkv -crf 18 -preset ultrafast out.mp4
 
Another approach could be to use gstreamer but I wanted to rely on simpler tools only. A slight disadvantage of the above approach is the temporary mkv file.
 
Best
Lars
 
 
On 2014-08-16 00:34 +0900, Konstantinos Theofilis wrote:
Hi list,
 
I have recorded logs from the robot in an experiment using dataDumper. At the same time, a video was taken of the experiment using a normal video camera.
 
I have a system that lets me know the yarp timestamp of one specific frame of the video acquired. E.g. I know that frame x_1 (not the first frame of the actual video file though) of the video file has a yarp timestamp of t_1. Subsequently, I know the timestamp t_n of the other frames x_n, by adding (1000/FPS)*(n-1) milliseconds to t_1.
 
What I want to do is to be able to replay the video in a synchronised fashion with dataSetPlayer playing the logs from the robot.
 
I suppose that one way is to use yarpdev ffpmeg to convert the video to a series of YarpImageOf bottles and record them with dataDumper. Then (with a script) modify the timestamps accordingly. But in that case,  as the video is 1080p, I am worried that any missing frames in the output of the driver will distort the timestamps. Videos are approximately 6-7 mins long.
 
Another way would be to convert the video file to ppm images, then adding the timestamps, since I know the timestamp t_1 of frame x_1, in a dataDumper compatible format.
 
My issue with both the above approaches is the amount of space needed, since I have over 100 videos.
 
Is there a reasonably accurate way to achieve what I want, without resorting to the "video->ppm->manually (script) add timestamps" process?
 
Best regards,
Kostas Theofilis
 
 
 
------------------------------------------------------------------------------
 
 
 
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
 
 
 
 
------------------------------------------------------------------------------
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers

 



------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

Lorenzo Natale-2

Ok. What is the datadumper doing with the timestamps when producing an avi? Is the avi respecting the original timestamps?

 

From: Konstantinos Theofilis [mailto:[hidden email]]
Sent: venerdì 29 agosto 2014 11:32
To: [hidden email]
Subject: [LIKELY_SPAM]Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

 

Hi all,

indeed, as Ugo says, dataDumper is already capable of producing an avi file. The only minor annoyance is that it needs (at least on debian/ubuntu systems) for ffmpeg and openCV to be compiled from source, which has more to do with the libav/ffmpeg situation in debian repositories and not dataDumper itself.

On the other hand, dataSet player cannot replay an avi file. That (for me at least) would be useful and I am in the process of implementing it.

Best,
Kostas Theofilis

On 29/08/14 10:23, Ugo Pattacini wrote:

Ciao Lorenzo,

 

Maybe I did not catch the point, but dataDumper is already able to produce an avi file with the options “--type video” or “--type image --addVideo”; the latter to produce both images and an avi.

 

Ciao,

Ugo

 

From: Lorenzo Natale [[hidden email]]
Sent: 29 August 2014 10:18
To: Konstantinos Theofilis; [hidden email]
Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

 

Dear Lars, Konstantinos,

Interesting discussion indeed.

 

> P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file?

 

The datadumper could dump directly an avi instead of individual frames. If somebody is willing to work on it, it would be very much appreciated.

 

Lorenzo

 

 

From: Konstantinos Theofilis [[hidden email]]
Sent: sabato 16 agosto 2014 21:16
To: [hidden email]
Subject: [LIKELY_SPAM]Re: [rc-hackers] synchronising videos with dataDumper logs

 

Hi Lars,

thanks for your comprehensive answer.

So by following your suggestion, I can produce a video file that incorporates the yarp timestamps (just a reminder, we are talking about the original video acquired by a normal video camera, not running through some YARP grabber).

However, it is not clear to me how i can play that video in a synchronised fashion with the logged data from dataDumper (without converting it beforehand to separate images).

E.g. is it possible to have dataSetPlayer to play the video (with the incorporated timestamps, or the timestamps in an external data.log file)?

Best regards,
Kostas Theofilis

P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file? Like: 1) Converting ppm images to an avi 2) Converting data.log to ts.list 3) mkvmerge the avi with ts.list 4) Final video with corrected timing info


On 16/08/14 16:17, Lars Schillingmann wrote:

Hi Konstantinos and all,
 
A while ago I also thought about that, and it seems indeed an issue for most video processing tools to process external presentation time stamps (PTS), etc.. The datadumper usually writes a file data.log and a file video.avi (HUFFYUV encoding, http://wiki.icub.org/iCub_documentation/group__dataDumper.html) if it is run with the corresponding options to save video data. The logfile contains the timestamps. Since you are anyway able to acquire the timestamps, I just mentioned it to be complete.
 
mkvmerge is able to process external timestamp information. I use the following python snippet to create a file that creates a compatible timestamp file from a datadumper data.log file:
print '# timecode format v2'
prevts = 0.0
for x in file('data.log'):
        ts = float(x.split()[1])
        if prevts == 0.0:
                prevts = ts
        print "%0.0f" % ((ts - prevts)*1000)
 
The timecode format is documented here:
https://www.bunkus.org/videotools/mkvtoolnix/doc/mkvmerge.html
(see 18.2)
 
Assuming that ts.list contains the timestamps in the format described above, mkvmerge can be called like this:
mkvmerge -o out.mkv --timecodes 0:ts.list video.avi
 
After that the mkv file contains the correct timing information for each frame. Since many tools cannot handle variable frame rates it should be processed with ffmpeg. ffmpeg's default behavior duplicates frames where necessary to achieve a constant frame rate. Some compression e.g. h264 also makes sense in most cases.
E.g. ffmpeg -i out.mkv -crf 18 -preset ultrafast out.mp4
 
Another approach could be to use gstreamer but I wanted to rely on simpler tools only. A slight disadvantage of the above approach is the temporary mkv file.
 
Best
Lars
 
 
On 2014-08-16 00:34 +0900, Konstantinos Theofilis wrote:
Hi list,
 
I have recorded logs from the robot in an experiment using dataDumper. At the same time, a video was taken of the experiment using a normal video camera.
 
I have a system that lets me know the yarp timestamp of one specific frame of the video acquired. E.g. I know that frame x_1 (not the first frame of the actual video file though) of the video file has a yarp timestamp of t_1. Subsequently, I know the timestamp t_n of the other frames x_n, by adding (1000/FPS)*(n-1) milliseconds to t_1.
 
What I want to do is to be able to replay the video in a synchronised fashion with dataSetPlayer playing the logs from the robot.
 
I suppose that one way is to use yarpdev ffpmeg to convert the video to a series of YarpImageOf bottles and record them with dataDumper. Then (with a script) modify the timestamps accordingly. But in that case,  as the video is 1080p, I am worried that any missing frames in the output of the driver will distort the timestamps. Videos are approximately 6-7 mins long.
 
Another way would be to convert the video file to ppm images, then adding the timestamps, since I know the timestamp t_1 of frame x_1, in a dataDumper compatible format.
 
My issue with both the above approaches is the amount of space needed, since I have over 100 videos.
 
Is there a reasonably accurate way to achieve what I want, without resorting to the "video->ppm->manually (script) add timestamps" process?
 
Best regards,
Kostas Theofilis
 
 
 
------------------------------------------------------------------------------
 
 
 
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
 
 
 
 
------------------------------------------------------------------------------
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers

 

 


------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

Konstantinos Theofilis
Hi Lorenzo,

when using the dataDumper with the --type video option, dataDumper produces a video avi file and a data.log file with timestamps.

The problem is that the produced avi file may not have a steady framerate, so the timestamps in the data.log are not corresponding to a steady framerate video.

The solution to this is the one Lars gave. I copy paste it here:

On 16/08/14 16:17, Lars Schillingmann wrote:

mkvmerge is able to process external timestamp information. I use the following python snippet to create a file that creates a compatible timestamp file from a datadumper data.log file:
print '# timecode format v2'
prevts = 0.0
for x in file('data.log'):
        ts = float(x.split()[1])
        if prevts == 0.0:
                prevts = ts
        print "%0.0f" % ((ts - prevts)*1000)

The timecode format is documented here:
https://www.bunkus.org/videotools/mkvtoolnix/doc/mkvmerge.html
(see 18.2)

Assuming that ts.list contains the timestamps in the format described above, mkvmerge can be called like this:
mkvmerge -o out.mkv --timecodes 0:ts.list video.avi

After that the mkv file contains the correct timing information for each frame. Since many tools cannot handle variable frame rates it should be processed with ffmpeg. ffmpeg's default behavior duplicates frames where necessary to achieve a constant frame rate. Some compression e.g. h264 also makes sense in most cases.
E.g. ffmpeg -i out.mkv -crf 18 -preset ultrafast out.mp4

Best,
Kostas Theofilis




On 29/08/14 11:20, Lorenzo Natale wrote:

Ok. What is the datadumper doing with the timestamps when producing an avi? Is the avi respecting the original timestamps?

 

From: Konstantinos Theofilis [[hidden email]]
Sent: venerdì 29 agosto 2014 11:32
To: [hidden email]
Subject: [LIKELY_SPAM]Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

 

Hi all,

indeed, as Ugo says, dataDumper is already capable of producing an avi file. The only minor annoyance is that it needs (at least on debian/ubuntu systems) for ffmpeg and openCV to be compiled from source, which has more to do with the libav/ffmpeg situation in debian repositories and not dataDumper itself.

On the other hand, dataSet player cannot replay an avi file. That (for me at least) would be useful and I am in the process of implementing it.

Best,
Kostas Theofilis

On 29/08/14 10:23, Ugo Pattacini wrote:

Ciao Lorenzo,

 

Maybe I did not catch the point, but dataDumper is already able to produce an avi file with the options “--type video” or “--type image --addVideo”; the latter to produce both images and an avi.

 

Ciao,

Ugo

 

From: Lorenzo Natale [[hidden email]]
Sent: 29 August 2014 10:18
To: Konstantinos Theofilis; [hidden email]
Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

 

Dear Lars, Konstantinos,

Interesting discussion indeed.

 

> P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file?

 

The datadumper could dump directly an avi instead of individual frames. If somebody is willing to work on it, it would be very much appreciated.

 

Lorenzo

 

 

From: Konstantinos Theofilis [[hidden email]]
Sent: sabato 16 agosto 2014 21:16
To: [hidden email]
Subject: [LIKELY_SPAM]Re: [rc-hackers] synchronising videos with dataDumper logs

 

Hi Lars,

thanks for your comprehensive answer.

So by following your suggestion, I can produce a video file that incorporates the yarp timestamps (just a reminder, we are talking about the original video acquired by a normal video camera, not running through some YARP grabber).

However, it is not clear to me how i can play that video in a synchronised fashion with the logged data from dataDumper (without converting it beforehand to separate images).

E.g. is it possible to have dataSetPlayer to play the video (with the incorporated timestamps, or the timestamps in an external data.log file)?

Best regards,
Kostas Theofilis

P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file? Like: 1) Converting ppm images to an avi 2) Converting data.log to ts.list 3) mkvmerge the avi with ts.list 4) Final video with corrected timing info


On 16/08/14 16:17, Lars Schillingmann wrote:

Hi Konstantinos and all,
 
A while ago I also thought about that, and it seems indeed an issue for most video processing tools to process external presentation time stamps (PTS), etc.. The datadumper usually writes a file data.log and a file video.avi (HUFFYUV encoding, http://wiki.icub.org/iCub_documentation/group__dataDumper.html) if it is run with the corresponding options to save video data. The logfile contains the timestamps. Since you are anyway able to acquire the timestamps, I just mentioned it to be complete.
 
mkvmerge is able to process external timestamp information. I use the following python snippet to create a file that creates a compatible timestamp file from a datadumper data.log file:
print '# timecode format v2'
prevts = 0.0
for x in file('data.log'):
        ts = float(x.split()[1])
        if prevts == 0.0:
                prevts = ts
        print "%0.0f" % ((ts - prevts)*1000)
 
The timecode format is documented here:
https://www.bunkus.org/videotools/mkvtoolnix/doc/mkvmerge.html
(see 18.2)
 
Assuming that ts.list contains the timestamps in the format described above, mkvmerge can be called like this:
mkvmerge -o out.mkv --timecodes 0:ts.list video.avi
 
After that the mkv file contains the correct timing information for each frame. Since many tools cannot handle variable frame rates it should be processed with ffmpeg. ffmpeg's default behavior duplicates frames where necessary to achieve a constant frame rate. Some compression e.g. h264 also makes sense in most cases.
E.g. ffmpeg -i out.mkv -crf 18 -preset ultrafast out.mp4
 
Another approach could be to use gstreamer but I wanted to rely on simpler tools only. A slight disadvantage of the above approach is the temporary mkv file.
 
Best
Lars
 
 
On 2014-08-16 00:34 +0900, Konstantinos Theofilis wrote:
Hi list,
 
I have recorded logs from the robot in an experiment using dataDumper. At the same time, a video was taken of the experiment using a normal video camera.
 
I have a system that lets me know the yarp timestamp of one specific frame of the video acquired. E.g. I know that frame x_1 (not the first frame of the actual video file though) of the video file has a yarp timestamp of t_1. Subsequently, I know the timestamp t_n of the other frames x_n, by adding (1000/FPS)*(n-1) milliseconds to t_1.
 
What I want to do is to be able to replay the video in a synchronised fashion with dataSetPlayer playing the logs from the robot.
 
I suppose that one way is to use yarpdev ffpmeg to convert the video to a series of YarpImageOf bottles and record them with dataDumper. Then (with a script) modify the timestamps accordingly. But in that case,  as the video is 1080p, I am worried that any missing frames in the output of the driver will distort the timestamps. Videos are approximately 6-7 mins long.
 
Another way would be to convert the video file to ppm images, then adding the timestamps, since I know the timestamp t_1 of frame x_1, in a dataDumper compatible format.
 
My issue with both the above approaches is the amount of space needed, since I have over 100 videos.
 
Is there a reasonably accurate way to achieve what I want, without resorting to the "video->ppm->manually (script) add timestamps" process?
 
Best regards,
Kostas Theofilis
 
 
 
------------------------------------------------------------------------------
 
 
 
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
 
 
 
 
------------------------------------------------------------------------------
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers

 

 



------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

pattacini
In reply to this post by Lorenzo Natale-2

Currently, the avi file can be saved with only fixed frame rate information: dataDumper makes use of the VideoWriter class of OpenCV to achieve that.

Therefore, at beginning of the acquisition the fps is extracted from the incoming stream of images and the avi is open with that parameter.

This means that, during the acquisition, the fluctuation of fps are not considered and it’s likely that the sync is lost with the motor data (even if we speak about of milliseconds). The video was intended indeed to provide user with a “summary”, while images carry the time information.

 

I don’t know if there exists a smarter interface that can associate a relative timestamp to the individual frame in an avi file (wrt the starting time instant) and I don’t know if the avi format allows for this. If so, we can try to implement it.

 

Nonetheless, I think Kostas is more focusing on the possibility to let dataSetPlayer replay a video instead of a set of images (bearing in mind the above caveat on the sync).

 

Ugo

 

From: Lorenzo Natale [mailto:[hidden email]]
Sent: 29 August 2014 12:21
To: Konstantinos Theofilis; [hidden email]
Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

 

Ok. What is the datadumper doing with the timestamps when producing an avi? Is the avi respecting the original timestamps?

 

From: Konstantinos Theofilis [[hidden email]]
Sent: venerdì 29 agosto 2014 11:32
To: [hidden email]
Subject: [LIKELY_SPAM]Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

 

Hi all,

indeed, as Ugo says, dataDumper is already capable of producing an avi file. The only minor annoyance is that it needs (at least on debian/ubuntu systems) for ffmpeg and openCV to be compiled from source, which has more to do with the libav/ffmpeg situation in debian repositories and not dataDumper itself.

On the other hand, dataSet player cannot replay an avi file. That (for me at least) would be useful and I am in the process of implementing it.

Best,
Kostas Theofilis

On 29/08/14 10:23, Ugo Pattacini wrote:

Ciao Lorenzo,

 

Maybe I did not catch the point, but dataDumper is already able to produce an avi file with the options “--type video” or “--type image --addVideo”; the latter to produce both images and an avi.

 

Ciao,

Ugo

 

From: Lorenzo Natale [[hidden email]]
Sent: 29 August 2014 10:18
To: Konstantinos Theofilis; [hidden email]
Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

 

Dear Lars, Konstantinos,

Interesting discussion indeed.

 

> P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file?

 

The datadumper could dump directly an avi instead of individual frames. If somebody is willing to work on it, it would be very much appreciated.

 

Lorenzo

 

 

From: Konstantinos Theofilis [[hidden email]]
Sent: sabato 16 agosto 2014 21:16
To: [hidden email]
Subject: [LIKELY_SPAM]Re: [rc-hackers] synchronising videos with dataDumper logs

 

Hi Lars,

thanks for your comprehensive answer.

So by following your suggestion, I can produce a video file that incorporates the yarp timestamps (just a reminder, we are talking about the original video acquired by a normal video camera, not running through some YARP grabber).

However, it is not clear to me how i can play that video in a synchronised fashion with the logged data from dataDumper (without converting it beforehand to separate images).

E.g. is it possible to have dataSetPlayer to play the video (with the incorporated timestamps, or the timestamps in an external data.log file)?

Best regards,
Kostas Theofilis

P.S. By the way, is the process you mentioned working also for converting the ppm files collected from dataDumper to a properly timestamped video file? Like: 1) Converting ppm images to an avi 2) Converting data.log to ts.list 3) mkvmerge the avi with ts.list 4) Final video with corrected timing info



On 16/08/14 16:17, Lars Schillingmann wrote:

Hi Konstantinos and all,
 
A while ago I also thought about that, and it seems indeed an issue for most video processing tools to process external presentation time stamps (PTS), etc.. The datadumper usually writes a file data.log and a file video.avi (HUFFYUV encoding, http://wiki.icub.org/iCub_documentation/group__dataDumper.html) if it is run with the corresponding options to save video data. The logfile contains the timestamps. Since you are anyway able to acquire the timestamps, I just mentioned it to be complete.
 
mkvmerge is able to process external timestamp information. I use the following python snippet to create a file that creates a compatible timestamp file from a datadumper data.log file:
print '# timecode format v2'
prevts = 0.0
for x in file('data.log'):
        ts = float(x.split()[1])
        if prevts == 0.0:
                prevts = ts
        print "%0.0f" % ((ts - prevts)*1000)
 
The timecode format is documented here:
https://www.bunkus.org/videotools/mkvtoolnix/doc/mkvmerge.html
(see 18.2)
 
Assuming that ts.list contains the timestamps in the format described above, mkvmerge can be called like this:
mkvmerge -o out.mkv --timecodes 0:ts.list video.avi
 
After that the mkv file contains the correct timing information for each frame. Since many tools cannot handle variable frame rates it should be processed with ffmpeg. ffmpeg's default behavior duplicates frames where necessary to achieve a constant frame rate. Some compression e.g. h264 also makes sense in most cases.
E.g. ffmpeg -i out.mkv -crf 18 -preset ultrafast out.mp4
 
Another approach could be to use gstreamer but I wanted to rely on simpler tools only. A slight disadvantage of the above approach is the temporary mkv file.
 
Best
Lars
 
 
On 2014-08-16 00:34 +0900, Konstantinos Theofilis wrote:
Hi list,
 
I have recorded logs from the robot in an experiment using dataDumper. At the same time, a video was taken of the experiment using a normal video camera.
 
I have a system that lets me know the yarp timestamp of one specific frame of the video acquired. E.g. I know that frame x_1 (not the first frame of the actual video file though) of the video file has a yarp timestamp of t_1. Subsequently, I know the timestamp t_n of the other frames x_n, by adding (1000/FPS)*(n-1) milliseconds to t_1.
 
What I want to do is to be able to replay the video in a synchronised fashion with dataSetPlayer playing the logs from the robot.
 
I suppose that one way is to use yarpdev ffpmeg to convert the video to a series of YarpImageOf bottles and record them with dataDumper. Then (with a script) modify the timestamps accordingly. But in that case,  as the video is 1080p, I am worried that any missing frames in the output of the driver will distort the timestamps. Videos are approximately 6-7 mins long.
 
Another way would be to convert the video file to ppm images, then adding the timestamps, since I know the timestamp t_1 of frame x_1, in a dataDumper compatible format.
 
My issue with both the above approaches is the amount of space needed, since I have over 100 videos.
 
Is there a reasonably accurate way to achieve what I want, without resorting to the "video->ppm->manually (script) add timestamps" process?
 
Best regards,
Kostas Theofilis
 
 
 
------------------------------------------------------------------------------
 
 
 
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
 
 
 
 
------------------------------------------------------------------------------
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers

 

 


------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

Lars Schillingmann
In reply to this post by Lorenzo Natale-2
Hi all,

No it is not but the timestamps are logged to data.log, so they can be associated by frame number. That is why I sometimes use mkvmerge to achieve correct timing. Btw avi is an old container format (it has very limited capabilities). Using mkv or mp4 would be better. A big issue is that one is relatively limited in producing flexible video formats withing opencv, and keeping the code platform independent at the same time.

Best
Lars

On 2014-08-29 19:20 +0900, Lorenzo Natale wrote:

> Ok. What is the datadumper doing with the timestamps when producing an avi? Is the avi respecting the original timestamps?
>
>  
>
> *From:*Konstantinos Theofilis [mailto:[hidden email]]
> *Sent:* venerdì 29 agosto 2014 11:32
> *To:* [hidden email]
> *Subject:* [LIKELY_SPAM]Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising videos with dataDumper logs
>
>  
>
> Hi all,
>
> indeed, as Ugo says, dataDumper is already capable of producing an avi file. The only minor annoyance is that it needs (at least on debian/ubuntu systems) for ffmpeg and openCV to be compiled from source, which has more to do with the libav/ffmpeg situation in debian repositories and not dataDumper itself.
>
> On the other hand, dataSet player cannot replay an avi file. That (for me at least) would be useful and I am in the process of implementing it.
>
> Best,
> Kostas Theofilis

 

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

pattacini
Yes Lars, indeed, good point. I forgot the data.log file :)
Anyway, are mp4 or mkv formats able to provide lossless data compression?
Vision developers might be concerned about artifacts introduced by lossy compressors.

Ugo

> -----Original Message-----
> From: Lars Schillingmann [mailto:[hidden email]]
> Sent: 29 August 2014 12:49
> To: [hidden email]
> Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising
> videos with dataDumper logs
>
> Hi all,
>
> No it is not but the timestamps are logged to data.log, so they can be
> associated by frame number. That is why I sometimes use mkvmerge to achieve
> correct timing. Btw avi is an old container format (it has very limited
> capabilities). Using mkv or mp4 would be better. A big issue is that one is
> relatively limited in producing flexible video formats withing opencv, and
> keeping the code platform independent at the same time.
>
> Best
> Lars
>
> On 2014-08-29 19:20 +0900, Lorenzo Natale wrote:
> > Ok. What is the datadumper doing with the timestamps when producing an avi?
> Is the avi respecting the original timestamps?
> >
> >
> >
> > *From:*Konstantinos Theofilis [mailto:[hidden email]]
> > *Sent:* venerdì 29 agosto 2014 11:32
> > *To:* [hidden email]
> > *Subject:* [LIKELY_SPAM]Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising
> videos with dataDumper logs
> >
> >
> >
> > Hi all,
> >
> > indeed, as Ugo says, dataDumper is already capable of producing an avi
> file. The only minor annoyance is that it needs (at least on debian/ubuntu
> systems) for ffmpeg and openCV to be compiled from source, which has more to
> do with the libav/ffmpeg situation in debian repositories and not dataDumper
> itself.
> >
> > On the other hand, dataSet player cannot replay an avi file. That (for me
> at least) would be useful and I am in the process of implementing it.
> >
> > Best,
> > Kostas Theofilis
>
>
>
> -----------------------------------------------------------------------------
> -
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Robotcub-hackers mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/robotcub-hackers

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

Lars Schillingmann
Hi Ugo

Yes, mkv and mp4 are only container formats in which you can put any number of video and audio tracks. Each video and audio track can have different codecs. I.e. it is no problem to produce a mkv file that contains a huffyuv (lossless, same as you are using in the dataDumper) compressed video track.

The difference to avi is that mkv and mp4 are modern, e.g. they support odd resolutions, etc.
A set of limitations of avi can be found here:
http://en.wikipedia.org/wiki/Audio_Video_Interleave#Limitations

Best
Lars
 

On 2014-08-29 19:56 +0900, Ugo Pattacini wrote:

> Yes Lars, indeed, good point. I forgot the data.log file :)
> Anyway, are mp4 or mkv formats able to provide lossless data compression?
> Vision developers might be concerned about artifacts introduced by lossy compressors.
>
> Ugo
>
>> -----Original Message-----
>> From: Lars Schillingmann [mailto:[hidden email]]
>> Sent: 29 August 2014 12:49
>> To: [hidden email]
>> Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising
>> videos with dataDumper logs
>>
>> Hi all,
>>
>> No it is not but the timestamps are logged to data.log, so they can be
>> associated by frame number. That is why I sometimes use mkvmerge to achieve
>> correct timing. Btw avi is an old container format (it has very limited
>> capabilities). Using mkv or mp4 would be better. A big issue is that one is
>> relatively limited in producing flexible video formats withing opencv, and
>> keeping the code platform independent at the same time.
>>
>> Best
>> Lars
>>
>> On 2014-08-29 19:20 +0900, Lorenzo Natale wrote:
>> > Ok. What is the datadumper doing with the timestamps when producing an avi?
>> Is the avi respecting the original timestamps?
>> >
>> >
>> >
>> > *From:*Konstantinos Theofilis [mailto:[hidden email]]
>> > *Sent:* venerdì 29 agosto 2014 11:32
>> > *To:* [hidden email]
>> > *Subject:* [LIKELY_SPAM]Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising
>> videos with dataDumper logs
>> >
>> >
>> >
>> > Hi all,
>> >
>> > indeed, as Ugo says, dataDumper is already capable of producing an avi
>> file. The only minor annoyance is that it needs (at least on debian/ubuntu
>> systems) for ffmpeg and openCV to be compiled from source, which has more to
>> do with the libav/ffmpeg situation in debian repositories and not dataDumper
>> itself.
>> >
>> > On the other hand, dataSet player cannot replay an avi file. That (for me
>> at least) would be useful and I am in the process of implementing it.
>> >
>> > Best,
>> > Kostas Theofilis
>>
>>
>>
>> -----------------------------------------------------------------------------
>> -
>> Slashdot TV.
>> Video for Nerds.  Stuff that matters.
>> http://tv.slashdot.org/
>> _______________________________________________
>> Robotcub-hackers mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
>

 

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

pattacini
Thanks Lars for the explanation.
Well, it seems that OpenCV is quite limited in supporting video container formats (see http://docs.opencv.org/doc/tutorials/highgui/video-write/video-write.html#the-structure-of-a-video).

I've tried with 'mp4' unsuccessfully, but 'mkv' worked on my windows platform.
If a deeper investigation on different platforms reveals that OpenCV provides a straightforward support for 'mkv' container, I think we can then switch to it.

Ciao,
Ugo

> -----Original Message-----
> From: Lars Schillingmann [mailto:[hidden email]]
> Sent: 29 August 2014 13:05
> To: Ugo Pattacini; [hidden email]
> Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising
> videos with dataDumper logs
>
> Hi Ugo
>
> Yes, mkv and mp4 are only container formats in which you can put any number
> of video and audio tracks. Each video and audio track can have different
> codecs. I.e. it is no problem to produce a mkv file that contains a huffyuv
> (lossless, same as you are using in the dataDumper) compressed video track.
>
> The difference to avi is that mkv and mp4 are modern, e.g. they support odd
> resolutions, etc.
> A set of limitations of avi can be found here:
> http://en.wikipedia.org/wiki/Audio_Video_Interleave#Limitations
>
> Best
> Lars
>
>
> On 2014-08-29 19:56 +0900, Ugo Pattacini wrote:
> > Yes Lars, indeed, good point. I forgot the data.log file :)
> > Anyway, are mp4 or mkv formats able to provide lossless data compression?
> > Vision developers might be concerned about artifacts introduced by lossy
> compressors.
> >
> > Ugo
> >
> >> -----Original Message-----
> >> From: Lars Schillingmann [mailto:[hidden email]]
> >> Sent: 29 August 2014 12:49
> >> To: [hidden email]
> >> Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising
> >> videos with dataDumper logs
> >>
> >> Hi all,
> >>
> >> No it is not but the timestamps are logged to data.log, so they can be
> >> associated by frame number. That is why I sometimes use mkvmerge to
> achieve
> >> correct timing. Btw avi is an old container format (it has very limited
> >> capabilities). Using mkv or mp4 would be better. A big issue is that one
> is
> >> relatively limited in producing flexible video formats withing opencv, and
> >> keeping the code platform independent at the same time.
> >>
> >> Best
> >> Lars
> >>
> >> On 2014-08-29 19:20 +0900, Lorenzo Natale wrote:
> >> > Ok. What is the datadumper doing with the timestamps when producing an
> avi?
> >> Is the avi respecting the original timestamps?
> >> >
> >> >
> >> >
> >> > *From:*Konstantinos Theofilis [mailto:[hidden email]]
> >> > *Sent:* venerdì 29 agosto 2014 11:32
> >> > *To:* [hidden email]
> >> > *Subject:* [LIKELY_SPAM]Re: [rc-hackers] [LIKELY_SPAM]Re: synchronising
> >> videos with dataDumper logs
> >> >
> >> >
> >> >
> >> > Hi all,
> >> >
> >> > indeed, as Ugo says, dataDumper is already capable of producing an avi
> >> file. The only minor annoyance is that it needs (at least on debian/ubuntu
> >> systems) for ffmpeg and openCV to be compiled from source, which has more
> to
> >> do with the libav/ffmpeg situation in debian repositories and not
> dataDumper
> >> itself.
> >> >
> >> > On the other hand, dataSet player cannot replay an avi file. That (for
> me
> >> at least) would be useful and I am in the process of implementing it.
> >> >
> >> > Best,
> >> > Kostas Theofilis
> >>
> >>
> >>
> >> --------------------------------------------------------------------------
> ---
> >> -
> >> Slashdot TV.
> >> Video for Nerds.  Stuff that matters.
> >> http://tv.slashdot.org/
> >> _______________________________________________
> >> Robotcub-hackers mailing list
> >> [hidden email]
> >> https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
> >
>
>

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
Reply | Threaded
Open this post in threaded view
|

Re: [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising videos with dataDumper logs

pattacini
In this respect, if we decide to switch to 'mkv' format (btw I've tested it on other platforms and it seems ok), I could make dataDumper produce the 'ts.list' file automatically in order to then easy the call to 'mkvmerge' tool, as Lars pointed out in a previous email, skipping the python step.

The final mkv file will contain the correct timestamps (i.e. with variable frame rate).

Ugo

> -----Original Message-----
> From: Ugo Pattacini [mailto:[hidden email]]
> Sent: 29 August 2014 13:18
> To: Lars Schillingmann; [hidden email]
> Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: [LIKELY_SPAM]Re: synchronising
> videos with dataDumper logs
>
> Thanks Lars for the explanation.
> Well, it seems that OpenCV is quite limited in supporting video container
> formats (see http://docs.opencv.org/doc/tutorials/highgui/video-write/video-
> write.html#the-structure-of-a-video).
>
> I've tried with 'mp4' unsuccessfully, but 'mkv' worked on my windows
> platform.
> If a deeper investigation on different platforms reveals that OpenCV provides
> a straightforward support for 'mkv' container, I think we can then switch to
> it.
>
> Ciao,
> Ugo
>
> > -----Original Message-----
> > From: Lars Schillingmann [mailto:[hidden email]]
> > Sent: 29 August 2014 13:05
> > To: Ugo Pattacini; [hidden email]
> > Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: [LIKELY_SPAM]Re:
> > synchronising videos with dataDumper logs
> >
> > Hi Ugo
> >
> > Yes, mkv and mp4 are only container formats in which you can put any
> > number of video and audio tracks. Each video and audio track can have
> > different codecs. I.e. it is no problem to produce a mkv file that
> > contains a huffyuv (lossless, same as you are using in the dataDumper)
> compressed video track.
> >
> > The difference to avi is that mkv and mp4 are modern, e.g. they
> > support odd resolutions, etc.
> > A set of limitations of avi can be found here:
> > http://en.wikipedia.org/wiki/Audio_Video_Interleave#Limitations
> >
> > Best
> > Lars
> >
> >
> > On 2014-08-29 19:56 +0900, Ugo Pattacini wrote:
> > > Yes Lars, indeed, good point. I forgot the data.log file :) Anyway,
> > > are mp4 or mkv formats able to provide lossless data compression?
> > > Vision developers might be concerned about artifacts introduced by
> > > lossy
> > compressors.
> > >
> > > Ugo
> > >
> > >> -----Original Message-----
> > >> From: Lars Schillingmann [mailto:[hidden email]]
> > >> Sent: 29 August 2014 12:49
> > >> To: [hidden email]
> > >> Subject: Re: [rc-hackers] [LIKELY_SPAM]Re: [LIKELY_SPAM]Re:
> > >> synchronising videos with dataDumper logs
> > >>
> > >> Hi all,
> > >>
> > >> No it is not but the timestamps are logged to data.log, so they can
> > >> be associated by frame number. That is why I sometimes use mkvmerge
> > >> to
> > achieve
> > >> correct timing. Btw avi is an old container format (it has very
> > >> limited capabilities). Using mkv or mp4 would be better. A big
> > >> issue is that one
> > is
> > >> relatively limited in producing flexible video formats withing
> > >> opencv, and keeping the code platform independent at the same time.
> > >>
> > >> Best
> > >> Lars
> > >>
> > >> On 2014-08-29 19:20 +0900, Lorenzo Natale wrote:
> > >> > Ok. What is the datadumper doing with the timestamps when
> > >> > producing an
> > avi?
> > >> Is the avi respecting the original timestamps?
> > >> >
> > >> >
> > >> >
> > >> > *From:*Konstantinos Theofilis [mailto:[hidden email]]
> > >> > *Sent:* venerdì 29 agosto 2014 11:32
> > >> > *To:* [hidden email]
> > >> > *Subject:* [LIKELY_SPAM]Re: [rc-hackers] [LIKELY_SPAM]Re:
> > >> > synchronising
> > >> videos with dataDumper logs
> > >> >
> > >> >
> > >> >
> > >> > Hi all,
> > >> >
> > >> > indeed, as Ugo says, dataDumper is already capable of producing
> > >> > an avi
> > >> file. The only minor annoyance is that it needs (at least on
> > >> debian/ubuntu
> > >> systems) for ffmpeg and openCV to be compiled from source, which
> > >> has more
> > to
> > >> do with the libav/ffmpeg situation in debian repositories and not
> > dataDumper
> > >> itself.
> > >> >
> > >> > On the other hand, dataSet player cannot replay an avi file. That
> > >> > (for
> > me
> > >> at least) would be useful and I am in the process of implementing it.
> > >> >
> > >> > Best,
> > >> > Kostas Theofilis
> > >>
> > >>
> > >>
> > >> -------------------------------------------------------------------
> > >> -------
> > ---
> > >> -
> > >> Slashdot TV.
> > >> Video for Nerds.  Stuff that matters.
> > >> http://tv.slashdot.org/
> > >> _______________________________________________
> > >> Robotcub-hackers mailing list
> > >> [hidden email]
> > >> https://lists.sourceforge.net/lists/listinfo/robotcub-hackers
> > >
> >
> >
>
> -----------------------------------------------------------------------------
> -
> Slashdot TV.
> Video for Nerds.  Stuff that matters.
> http://tv.slashdot.org/
> _______________________________________________
> Robotcub-hackers mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/robotcub-hackers

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Robotcub-hackers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/robotcub-hackers