CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Column-based storage for analysis of high-frequency stock trading data
KTH, School of Electrical Engineering and Computer Science (EECS).
2019 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE creditsStudent thesis
Abstract [en]

This study investigated the efficiency of the available open-source columnbased storage formats with support for semi-flexible data in combination with query engines that support querying these formats. Two different formats were identified, Parquet and ORC, and both were tested in two different modes, uncompressed and compressed with the compression algorithm Snappy. They were tested by running two queries on the host company’s data converted to the appropriate formats, one simple averaging query and one more complicated with counts and filtering. The queries were run with two different query engines, Spark and Drill. They were also run on two dataset with different sizes to test scalability. The query execution time was recorded for each tested alternative. The results show that Snappy compressed formats always outperformed their non-compressed counterparts, and that Parquet was always faster than ORC. Drill performed faster on the simple query while Spark performed faster on the complex query. Drill also had the least increase in query execution time when the size of the dataset increased on both queries. The conclusion is that Parquet with Snappy is the storage format which gives the fastest execution times. However, both Spark and Drill have their own advantages as query engines.

Abstract [sv]

Denna studie undersökte effektiviteten av de i öppen källkod tillgängliga kolumnbaserade lagringsformaten med stöd för semistrukturerad data i kombination med frågemotorer som stödjer dessa format. Två olika format identifierades, Parquet och ORC, och båda testades i på olika sätt, okomprimerade och komprimerade med kompressionsalgoritmen Snappy. De testades genom att köra två frågor på uppdragsgivarens data som konverterades till de testade formaten, en enkel som räknar genomsnitt och en mer komplicerad med radräkning och filtrering. Båda frågorna kördes med två olika frågemotorer, Spark and Drill. De kördes på två datamängder med olika storlekar för att testa skalbarhet. Exekveringstiden mättes för varje testat alternativ. Resultaten visar att Snappy-komprimerade format alltid exekverade snabbare än de ickekomprimerade formaten, och att Parquet alltid körde snabbare än ORC. Drill var snabbare på den enkla frågan medan Spark var snabbare på den komplexa. Drill hade också den minsta ökningen i exekveringstiden när storleken på datamängden ökade på båda frågorna. Slutsatsen är att Parquet med Snappy är det lagringsformat som ger den snabbaste exekveringstiden, och att både Spark och Drill har sina egna fördelar.

Place, publisher, year, edition, pages
2019. , p. 39
Series
TRITA-EECS-EX ; 2019:484
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:kth:diva-255019OAI: oai:DiVA.org:kth-255019DiVA, id: diva2:1337268
Examiners
Available from: 2019-07-12 Created: 2019-07-12 Last updated: 2019-07-12Bibliographically approved

Open Access in DiVA

fulltext(1097 kB)9 downloads
File information
File name FULLTEXT01.pdfFile size 1097 kBChecksum SHA-512
50547954ac33ba8e02a17b22950f3f4406cd6559682fb27ac5f19cc44d0e6fbb31a2711fed93d591b5b03bd2b2421c38a6c123f003f8f0224dab9384c6302dde
Type fulltextMimetype application/pdf

By organisation
School of Electrical Engineering and Computer Science (EECS)
Computer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 9 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 18 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf