Database dump survey

Started by byte[]
Posted
139 replies
Login to subscribe to responses
JP's avatar
Posts: 1578
JP
The End wasn't The End - Found a new home after the great exodus of 2012
Magnificent Metadata Maniac - #1 Assistant
Friendship, Art, and Magic (5 Years) - Celebrated Derpibooru's five year anniversary with friends.
Not a Llama - Happy April Fools Day!
Birthday Cake - Celebrated MLP's 7th birthday
Cool Crow - "Caw!" An awesome tagger
Friendship, Art, and Magic (6 Years) - Celebrated Derpibooru's six year anniversary with friends.
A Tale For The Ages - Celebrated MLP's 35th Anniversary and FiM's 8th Anniversary
Artist
Friendship, Art, and Magic (7 Years) - Celebrated Derpibooru's seventh year anniversary with friends.
Best Artist - Providing quality, Derpibooru-exclusive artwork
Wallet After Summer Sale

When can we expect the first dump to be released? I’m itching to do some tag-related statistics stuff (like what tags I’m most often adding to images).

>inb4 tag change histories are not part of the dump :-(

Also a question about the dump format. The "PostgreSQL custom dump format" part worries me a bit. AFAIK PGSQL dumps can be converted into sqlite dumps, but if it’s a "custom" format, then I don’t know.
Posted Report
Background Pony #99CD
Sorry, maybe I’m asking the wrong place.
Does this dump include all uploaded images?
Is there already such a dump?
Posted Report
byte[]'s avatar
Posts: 1723
byte[]
Site Administrator
Site Developer
@JP
When can we expect the first dump to be released? I’m itching to do some tag-related statistics stuff (like what tags I’m most often adding to images).

Probably a few months. I’m still working out the gremlins in the schema structure. I also need some time to verify that the dumps we make contain everything users wanted and don’t leak any private information.

>inb4 tag change histories are not part of the dump :-(

It was requested, so I’ll add them.

Also a question about the dump format. The "PostgreSQL custom dump format" part worries me a bit. AFAIK PGSQL dumps can be converted into sqlite dumps, but if it’s a "custom" format, then I don’t know.

See the documentation for pg_dump. The custom dump format, despite being non-portable, is the most flexible, because you can choose which tables you want to restore when you load it, and you can restore in parallel (assuming it wouldn’t swamp your drive). It’s also the fastest for me to make, since I can run it in parallel. You can’t do either of these things with the SQL format dump.
Posted Report
mjangelvortex's avatar
Posts: 7516
mjangelvortex
Equality - In our state, we do not stand out.
Artist
A Tale For The Ages - Celebrated MLP's 35th Anniversary and FiM's 8th Anniversary
Magnificent Metadata Maniac - #1 Assistant
Friendship, Art, and Magic (7 Years) - Celebrated Derpibooru's seventh year anniversary with friends.
Wallet After Summer Sale
Toola Roola - For helping others attend the 2019 Community Collab
Notoriously Divine Tagger - Consistently uploads images above and beyond the minimum tag requirements. And/or additionally, bringing over the original description from the source if the image has one. Does NOT apply to the uploader adding several to a dozen tags after originally uploading with minimum to bare tagging.

Lady of Ships and Birbs
@ZizzyDizzyMC
I wouldn’t mind to be able to see some of my deleted comments. There were two images here that were deleted recently that I commented on that I wouldn’t mind being able to see.
Posted Report
byte[]'s avatar
Posts: 1723
byte[]
Site Administrator
Site Developer
Preliminary testing shows that the public dump should be about 2.5GB, and it takes about 25 minutes to generate on our side. (This doesn’t require any downtime.)
Posted Report
Ralek's avatar
Posts: 80
Ralek
Sigma Butt
Smile - Derpi Supporter
Daring Do Dakimakura - Attended a Derpibooru panel at a MLP convention
Happy Derpy!
Artist
Perfect Pony Plot Provider - 10+ uploads with over 350 upvotes or more (Questionable/Explicit)
Hard Work - Merited Perfect Pony Plot Provider badge with only their own art
Responsible Disclosure
An Artist Who Rocks - 100+ images under their artist tag
A Really Classy Artist - 250+ images under their artist tag
A Really Hyper Artist - 500+ images under their artist tag
Best Artist - Providing quality, Derpibooru-exclusive artwork
Flexible - Bendy Pony
The Power of Love - Made a piece of artwork loved by 1000 users
Magnificence - Artist with 100+ Safe/Suggestive images with a score of over a 100, and 5+ Questionable/Explicit images with a score of over 350
Friendship, Art, and Magic (5 Years) - Celebrated Derpibooru's five year anniversary with friends.
Not a Llama - Happy April Fools Day!
Element of Generosity - For users who have donated to the site
Cock-a-doodle do - Raises every morning
Birthday Cake - Celebrated MLP's 7th birthday
Helpful Owl - Drew someone's OC for the 2018 Community Collab
Friendship, Art, and Magic (6 Years) - Celebrated Derpibooru's six year anniversary with friends.
Chaotic Little Trees - 1000+ images under their artist tag
Equality - In our state, we do not stand out.
Lady's Wink
Fine Arts - Two hundred uploads with a score of over a hundred (Safe/Suggestive)
Gold Patron - Gold Patron
Notoriously Divine Tagger - Consistently uploads images above and beyond the minimum tag requirements. And/or additionally, bringing over the original description from the source if the image has one. Does NOT apply to the uploader adding several to a dozen tags after originally uploading with minimum to bare tagging.
A Tale For The Ages - Celebrated MLP's 35th Anniversary and FiM's 8th Anniversary
Silver Bit
Heart Gem
Wallet After Summer Sale

Smile
This would definitely be useful, I know I’ve used the API to grab large chunks of data before.
If it hasn’t been suggested, maybe publicize your method of reverse image searches and include the hashes in the dumps?
Posted Report
Background Pony #DD72
If I right understood, with this dump any developer could use Derpibooru archive directly, without user interface?

We want create our own Derpibooru interface – with modern design and russian language support (I’m Brony Webmaster from Russia, yes). Also, standard Derpibooru interface is blocked in Russia (because foalcon denied in our country, same as child porn), but DerpiCDN is available – so we may make our interface with totally foalcon excluding, to avoid blocking it (or may be with hidden cheatcode or premium account to unlock foalcon).

That database dumps probably much helps us to create our custom Derpibooru interface. Good luck!
Posted Report
byte[]'s avatar
Posts: 1723
byte[]
Site Administrator
Site Developer
@Background Pony #DD72
To be clear, these archives will not include any user data beyond what would normally be displayed by the site. That means if you want to use authentication, you’ll have to roll your own system.
Posted Report
Background Pony #67B5
Can you by any chance also include data that are made look like deleted (like data about deleted images, comments on deleted images, deleted comments, deleted forum posts and deletion reasons)?
Posted Report
Interested in advertising on Derpibooru? Click here for information!
Pony Arts & Prints!

Derpibooru costs over $25 a day to operate - help support us financially!

Syntax quick reference: *bold* _italic_ [spoiler]hide text[/spoiler] @code@ +underline+ -strike- ^sup^ ~sub~