Database dump survey

JP
My Little Pony - 1992 Edition
Friendship, Art, and Magic (2020) - Took part in the 2020 Community Collab
Dream Come True! - Participated in the MLP 9th Anniversary Event
Wallet After Summer Sale -
Best Artist - Providing quality, Derpibooru-exclusive artwork
Friendship, Art, and Magic (2019) - Celebrated Derpibooru's seventh year anniversary with friends.
Artist -
A Tale For The Ages - Celebrated MLP's 35th Anniversary and FiM's 8th Anniversary
Friendship, Art, and Magic (2018) - Celebrated Derpibooru's six year anniversary with friends.
Cool Crow - "Caw!" An awesome tagger

The magic's gone :-(
When can we expect the first dump to be released? I'm itching to do some tag-related statistics stuff (like what tags I'm most often adding to images).

>inb4 tag change histories are not part of the dump :-(

Also a question about the dump format. The "PostgreSQL custom dump format" part worries me a bit. AFAIK PGSQL dumps can be converted into sqlite dumps, but if it's a "custom" format, then I don't know.
Posted Report
Background Pony #B575
Sorry, maybe I'm asking the wrong place.
Does this dump include all uploaded images?
Is there already such a dump?
Posted Report
byte[]

Philomena Contributor
@JP
When can we expect the first dump to be released? I'm itching to do some tag-related statistics stuff (like what tags I'm most often adding to images).

Probably a few months. I'm still working out the gremlins in the schema structure. I also need some time to verify that the dumps we make contain everything users wanted and don't leak any private information.

>inb4 tag change histories are not part of the dump :-(

It was requested, so I'll add them.

Also a question about the dump format. The "PostgreSQL custom dump format" part worries me a bit. AFAIK PGSQL dumps can be converted into sqlite dumps, but if it's a "custom" format, then I don't know.

See the documentation for pg_dump. The custom dump format, despite being non-portable, is the most flexible, because you can choose which tables you want to restore when you load it, and you can restore in parallel (assuming it wouldn't swamp your drive). It's also the fastest for me to make, since I can run it in parallel. You can't do either of these things with the SQL format dump.
Posted Report
mjangelvortex
My Little Pony - 1992 Edition
Friendship, Art, and Magic (2020) - Took part in the 2020 Community Collab
The Magic of Friendship Grows - For helping others attend the 2020 Community Collab
Cool Birb - "Caw!" An awesome tagger
Dream Come True! - Participated in the MLP 9th Anniversary Event
Notoriously Divine Tagger - Consistently uploads images above and beyond the minimum tag requirements. And/or additionally, bringing over the original description from the source if the image has one. Does NOT apply to the uploader adding several to a dozen tags after originally uploading with minimum to bare tagging.
Toola Roola - For helping others attend the 2019 Community Collab
Wallet After Summer Sale -
Friendship, Art, and Magic (2019) - Celebrated Derpibooru's seventh year anniversary with friends.
Magnificent Metadata Maniac - #1 Assistant

Lady of Ships and Birbs
@ZizzyDizzyMC
I wouldn't mind to be able to see some of my deleted comments. There were two images here that were deleted recently that I commented on that I wouldn't mind being able to see.
Posted Report
byte[]

Philomena Contributor
Preliminary testing shows that the public dump should be about 2.5GB, and it takes about 25 minutes to generate on our side. (This doesn't require any downtime.)
Posted Report
Ralek
Daring Do Dakimakura - Attended a Derpibooru panel at a MLP convention
Smile - Derpi Supporter
Sigma Butt -
My Little Pony - 1992 Edition
Wallet After Summer Sale -
Heart Gem -
A Tale For The Ages - Celebrated MLP's 35th Anniversary and FiM's 8th Anniversary
Notoriously Divine Tagger - Consistently uploads images above and beyond the minimum tag requirements. And/or additionally, bringing over the original description from the source if the image has one. Does NOT apply to the uploader adding several to a dozen tags after originally uploading with minimum to bare tagging.
Fine Arts - Two hundred uploads with a score of over a hundred (Safe/Suggestive)
Lady's Wink -

Smile
This would definitely be useful, I know I've used the API to grab large chunks of data before.
If it hasn't been suggested, maybe publicize your method of reverse image searches and include the hashes in the dumps?
Posted Report
Background Pony #D733
If I right understood, with this dump any developer could use Derpibooru archive directly, without user interface?

We want create our own Derpibooru interface — with modern design and russian language support (I'm Brony Webmaster from Russia, yes). Also, standard Derpibooru interface is blocked in Russia (because foalcon denied in our country, same as child porn), but DerpiCDN is available — so we may make our interface with totally foalcon excluding, to avoid blocking it (or may be with hidden cheatcode or premium account to unlock foalcon).

That database dumps probably much helps us to create our custom Derpibooru interface. Good luck!
Posted Report
byte[]

Philomena Contributor
@Background Pony #DD72
To be clear, these archives will not include any user data beyond what would normally be displayed by the site. That means if you want to use authentication, you'll have to roll your own system.
Posted Report
Background Pony #2ECF
Can you by any chance also include data that are made look like deleted (like data about deleted images, comments on deleted images, deleted comments, deleted forum posts and deletion reasons)?
Posted Report
Interested in advertising on Derpibooru? Click here for information!
Furbooru - A furry-centric imageboard

Derpibooru costs over $25 a day to operate - help support us financially!

Syntax quick reference: *bold* _italic_ [spoiler]hide text[/spoiler] @code@ +underline+ -strike- ^sup^ ~sub~