Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Spatio-temporal scale selection in video data
KTH, School of Computer Science and Communication (CSC), Computational Science and Technology (CST). (Computational Brain Science Lab)ORCID iD: 0000-0002-9081-2170
2018 (English)In: Journal of Mathematical Imaging and Vision, ISSN 0924-9907, E-ISSN 1573-7683, Vol. 60, no 4, p. 525-562Article in journal (Refereed) Published
Abstract [en]

This work presents a theory and methodology for simultaneous detection of local spatial and temporal scales in video data. The underlying idea is that if we process video data by spatio-temporal receptive fields at multiple spatial and temporal scales, we would like to generate hypotheses about the spatial extent and the temporal duration of the underlying spatio-temporal image structures that gave rise to the feature responses.

For two types of spatio-temporal scale-space representations, (i) a non-causal Gaussian spatio-temporal scale space for offline analysis of pre-recorded video sequences and (ii) a time-causal and time-recursive spatio-temporal scale space for online analysis of real-time video streams, we express sufficient conditions for spatio-temporal feature detectors in terms of spatio-temporal receptive fields to deliver scale covariant and scale invariant feature responses.

We present an in-depth theoretical analysis of the scale selection properties of eight types of spatio-temporal interest point detectors in terms of either: (i)-(ii) the spatial Laplacian applied to the first- and second-order temporal derivatives, (iii)-(iv) the determinant of the spatial Hessian applied to the first- and second-order temporal derivatives, (v) the determinant of the spatio-temporal Hessian matrix, (vi) the spatio-temporal Laplacian and (vii)-(viii) the first- and second-order temporal derivatives of the determinant of the spatial Hessian matrix. It is shown that seven of these spatio-temporal feature detectors allow for provable scale covariance and scale invariance. Then, we describe a time-causal and time-recursive algorithm for detecting sparse spatio-temporal interest points from video streams and show that it leads to intuitively reasonable results.

An experimental quantification of the accuracy of the spatio-temporal scale estimates and the amount of temporal delay obtained these spatio-temporal interest point detectors is given showing that: (i) the spatial and temporal scale selection properties predicted by the continuous theory are well preserved in the discrete implementation and (ii) the spatial Laplacian or the determinant of the spatial Hessian applied to the first- and second-order temporal derivatives lead to much shorter temporal delays in a time-causal implementation compared to the determinant of the spatio-temporal Hessian or the first- and second-order temporal derivatives of the determinant of the spatial Hessian matrix.

Place, publisher, year, edition, pages
Springer Science+Business Media B.V., 2018. Vol. 60, no 4, p. 525-562
Keywords [en]
Scale selection, Scale, Scale space, Spatial, Temporal, Spatio-temporal, Scale invariance, Scale covariance, Feature detection, Differential invariant, Interest point, Video analysis, Computer vision
National Category
Computer Vision and Robotics (Autonomous Systems) Computer Sciences Mathematics
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:kth:diva-214932DOI: 10.1007/s10851-017-0766-9ISI: 000428612400004Scopus ID: 2-s2.0-85032330463OAI: oai:DiVA.org:kth-214932DiVA, id: diva2:1144192
Projects
Scale-space theory for invariant and covariant visual receptive fieldsTime-causal receptive fields for computer vision and computational modelling of biological vision
Funder
Swedish Research Council, 2014-4083Stiftelsen Olle Engkvist Byggmästare, 2015/465
Note

QC 20171129

Available from: 2017-09-25 Created: 2017-09-25 Last updated: 2018-12-06Bibliographically approved

Open Access in DiVA

fulltext(7775 kB)49 downloads
File information
File name FULLTEXT02.pdfFile size 7775 kBChecksum SHA-512
9b63eff807b0c0c65ac558f7e0ea155c84723ee6dad0e73084045446f7091d705541a759b19f566f3ab4b5f04691eea525893cca88c83ca6609e66bca97f2814
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Search in DiVA

By author/editor
Lindeberg, Tony
By organisation
Computational Science and Technology (CST)
In the same journal
Journal of Mathematical Imaging and Vision
Computer Vision and Robotics (Autonomous Systems)Computer SciencesMathematics

Search outside of DiVA

GoogleGoogle Scholar
Total: 51 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 654 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf