File Upload and Media Handling


1. environment

php.ini

  • upload_max_filesize
  • post_max_size
  • max_input_time
  • max_execution_time
  • max_file_uploads
  • session.gc_maxlifetime

storage

  1. file system
    1. global
    2. perpage
  2. database

1.1. settings

options

  1. upload
    1. [0, 1, Admins] – refactor, last value is actually just the allowed user group
  2. upload_max_size
  3. upload_quota
  4. upload_quota_per_user
  5. upload_banned_exts
  6. upload_images_only

  1. check_mimetype
  2. img_create_thumbnail
  3. img_max_thumb_width

constants

  1. define('THUMB_DIR', 'file/thumb');
  2. define('UPLOAD_GLOBAL_DIR', 'file/global');
  3. define('UPLOAD_PER_PAGE_DIR', 'file/perpage');

roles


permissions

  • $this->has_access('upload')

Anti-Hack

Download Flood Minutes:
The number of minutes to set flood control to.

Download Flood Control:

The number of downloads allowed in period of 'Download Flood Minutes'.

1.2. chmod

Write access for files folders.


  1. User can read, write and execute
  2. Group can read and execute
  3. World can read and execute

Execute privilege on directories is required to access files inside them. (AT&T)


The important thing is that the webserver has write access.


Thus if the files directory is owned by the webserver user ( e.g. www-data ) 755 would suffice.
If the webserver is not owner, but is in the group owning the directory, 775 would be better.
777 is the extreme case where all users on the system have full access. In most cases this would not be advisable.

1.3. MIME type and file extensions

https://en.wikipedia.org/wiki/Media_type
https://mimesniff.spec.whatwg.org/


Blacklisting File Extensions


Whitelisting File Extensions

# Allowed uploadable file extensions and mimetypes

$extension_map	= [
	// application
	'bz'		=> ['application', 'application/x-bzip'],
	'bz2'		=> ['application', 'application/x-bzip2'],
	'doc'		=> ['application', 'application/msword'],
	'exe'		=> ['application', 'application/octet-stream'],
	'latex'		=> ['application', 'application/x-latex'],
	'gtar'		=> ['application', 'application/x-gtar'],
	'gz'		=> ['application', 'application/x-gzip'],
	'gzip'		=> ['application', 'application/x-gzip'],
	'pdf'		=> ['application', 'application/pdf'],
	'ppt'		=> ['application', 'application/mspowerpoint'],
	'ps'		=> ['application', 'application/postscript'],
	'tar'		=> ['application', 'application/x-tar'],
	'tgz'		=> ['application', 'application/x-compressed'],
	'torrent'	=> ['application', 'application/x-bittorrent'],
	'xls'		=> ['application', 'application/excel'],
	'zip'		=> ['application', 'application/x-zip-compressed'],
	'7z'		=> ['application', 'application/x-7z-compressed'],

	'odc'		=> ['application', 'application/vnd.oasis.opendocument.chart'],
	'odb'		=> ['application', 'application/vnd.oasis.opendocument.database'],
	'odf'		=> ['application', 'application/vnd.oasis.opendocument.formula'],
	'odg'		=> ['application', 'application/vnd.oasis.opendocument.graphics'],
	'odi'		=> ['application', 'application/vnd.oasis.opendocument.image'],
	'odp'		=> ['application', 'application/vnd.oasis.opendocument.presentation'],
	'ods'		=> ['application', 'application/vnd.oasis.opendocument.spreadsheet'],
	'odt'		=> ['application', 'application/vnd.oasis.opendocument.text'],


	// (legacy)
	'pptx'		=> ['application', 'application/vnd.openxmlformats-officedocument.presentationml.presentation'],
	'xlsx'		=> ['application', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'],
	'docx'		=> ['application', 'application/vnd.openxmlformats-officedocument.wordprocessingml.document'],

	// audio
	'm4a'		=> ['audio', 'audio/mp4'],
	'mp3'		=> ['audio', 'audio/x-mpeg-3'],
	'ogg'		=> ['audio', 'audio/ogg'],
	'opus'		=> ['audio', 'audio/ogg'],

	// image
	'gif'		=> ['image', 'image/gif'],
	'ico'		=> ['image', 'image/x-icon'],
	'jpe'		=> ['image', 'image/jpeg'],
	'jpeg'		=> ['image', 'image/jpeg'],
	'jpg'		=> ['image', 'image/jpeg'],
	'png'		=> ['image', 'image/png'],
	'svg'		=> ['image', 'image/svg+xml'],
	'webp'		=> ['image', 'image/webp'],

	// text
	'conf'		=> ['text', 'text/plain'],
	'css'		=> ['text', 'text/css'],
	'htm'		=> ['text', 'text/html'],
	'html'		=> ['text', 'text/html'],
	'rtf'		=> ['text', 'text/richtext'],
	'txt'		=> ['text', 'text/plain'],
	'xml'		=> ['text', 'text/xml'],

	// video
	'mp4'		=> ['video', 'video/mp4'],
	'ogv'		=> ['video', 'video/ogg'],
	'webm'		=> ['video', 'video/webm'],
];

"Content-Type" Header Validation

1.4. HTML5


form

1.5. File APIs



Libs

1.6. Fetch API


Uploading files using 'fetch' and 'FormData'

1.7. PHP


$_FILES

  • $_FILES['uploadedfile']['name']: The original name of the file on the client machine
  • $_FILES['uploadedfile']['type']: The mime type of the file
  • $_FILES['uploadedfile']['size']: The size of the file in bytes
  • $_FILES['uploadedfile']['tmp_name']: The temporary filename in which the uploaded file was stored on the server.

functions

  1. mime_content_type
  2. pathinfo
  3. hash_file

1.8. Image processing


  1. GD Graphics Library
  2. ImageMagick

check the availability

if (extension_loaded('gd'))
{
    print_r(gd_info());
}
else
{
    echo 'GD is not available.';
}

if (extension_loaded('imagick'))
{
    $imagick = new Imagick();
    print_r($imagick->queryFormats());
}
else
{
    echo 'ImageMagick is not available.';
}

2. WackoWiki file model

2.1. File name sanitization


// prepare for translit
$name	= str_replace(['@', '%20', '+'], '-', $name);
$name	= preg_replace('/[\r\n\t -]+/', '_', $name);
$name	= trim($name, ' .-_');

// here would be place for translit
$t_name = $this->format($name, 'translit');

  1. global
    1. [file_name . extension]
    2. UPLOAD_GLOBAL_DIR
    3. direct file access
  2. local
    1. [@ . page_id . @ . file_name . extension]
    2. UPLOAD_PER_PAGE_DIR
    3. file access via file hadler
      1. checks page ACLs against user permissions

3. Database

  1. file_name
  2. author (user_id)
  3. file_description
  4. caption
  5. views
  6. uploaded_dt
  7. modified_dt
  8. credit
  9. exif
  10. context use (file_link)
  11. page_id
  12. categories
  13. access, viewing privacy
  14. file_lang
  15. file_size
  16. picture_w
  17. picture_h
  18. file_ext
  19. mime_type

4. Workflow


validation

  1. extension
  2. mime type
  3. size

header for file handler

"Content-Disposition: Attachment"
"X-Content-Type-Options: nosniff"

  1. https://developer.mozilla.org/[...]/Content-Disposition
  2. https://developer.mozilla.org/[...]Content-Type-Options

database

  1. mimetype (127+1+127 = 255)

https://stackoverflow.com/ques[...]n-storing-type-in-db


actions

  1. files
  2. upload

handler

  1. upload
  2. file
  3. attachments
  4. filemeta

functions

  • wacko -> link
  • wackoformatter

hacks


ideas

5. File Usage

Lotus Flower
Shinobazu Pond at Ueno Park at Ueno, Taito-ku in Tokyo
(Source: Yoshikazu TAKADA /Some rights reserved)

The file syntax is processed in the link function. Not to mention that the link function requires a refactoring.
Additional we have missing cases for <audio> and <video> tag, which needs to be added. DONE


In R5.5 a cation field was added to the file table.

<figure>
  <img src="https://wackowiki.org/doc/file/global/lotus_flower.jpg" alt="Lotus Flower" title="Lotus Flower (105 KiB)" width="640" height="427">
  <figcaption>Shinobazu Pond at Ueno Park at Ueno, Taito-ku in Tokyo. Original by <a href="https://www.flickr.com/photos/yoshikazut/">Yoshikazu TAKADA</a></figcaption>
</figure>

Make the width of <figcaption> match the width of the <img> inside its <figure> tag

figure { display: table; }
figcaption { display: table-caption; caption-side:bottom; } 

The file can be linked directly (file handler) or to filemeta handler.


Arguments can be added via ? and &


file:/image.png?200x400&direct&caption

5.1. Supported Media


Media Format
audio m4a, mp3, ogg, opus
image gif, jpg, jpe, jpeg, png, svg, webp
video mp4, ogv, webm

5.1.1. Audio

file:/audio.opus?caption

Johann Sebastian Bach - Well-Tempered Clavier, Book 1 - 01 Prelude No. 1 in C major, BWV 846
(Source: Composer: Johann Sebastian Bach; Performer: Kimiko Ishizaka /Public domain)

https://developer.mozilla.org/[...]b/HTML/Element/audio

5.1.2. Video

file:/video.webm?caption

Sandhill Cranes
(Source: NPS / Dale Bohlke /Public domain)

https://developer.mozilla.org/[...]b/HTML/Element/video
https://developer.mozilla.org/[...]itles_to_HTML5_video

5.2. Embedding

file:image.png
file:audio.opus
file:video.mp4

5.3. Media Parameters


img  audio video notes
linking
direct
nolink
linkonly
meta
alignment
right
left
center
resizing
400x200
600
others
caption
clear

5.3.1. Alignment

file:image.png?right


file:image.png?right&clear

5.3.2. Resizing

file:image.png?20x50


Lotus Flower
Shinobazu Pond at Ueno Park at Ueno, Taito-ku in Tokyo
(Source: Yoshikazu TAKADA /Some rights reserved)

5.3.3. Linking

file:image.png?direct
file:image.png?nolink
file:image.png?linkonly


By default, clicking on an image should brings up the filemeta handler page.


How we manage to link an image to another page with the file: syntax?

  • ((file:/image.png some text))
  • ((/Page/Link file:/image.png))

5.3.4. Caching

file:image.png?nocache

5.3.5. Others

file:image.png?caption


+ audio and video options

6. Bad Behavior strict mode issue

bb_settings.conf

strict = true

status_key: 7ad04a8a


HTTP load failed with status 400. Load of media resource https://wackowiki.org/doc/file/global/41265046264_720p.mp4 failed.
HTTP load failed with status 400. Load of media resource https://wackowiki.org/doc/file/global/kimiko_ishizaka___bach___well_tempered_clavier_book_1___01_prelude_no._1_in_c_major_bwv_846.ogg failed.


[400] The automated program you are using is not permitted to access this server. Please use a different program or a standard Web browser.
-> Prohibited header 'Range' present


Range: bytes=0-
Referer: https://wackowiki.org/doc/Dev/Release/R55/FileUpload
Accept-Language: en-US,en;q=0.5
Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0
Host: wackowiki.org
Mod-Rewrite: on
Mod-Env: on

Suppose you have a video or audio of 4M bytes, when your browser request for video the first time it will send headers like this

Host:localhost
Range:bytes=0-

Range header bytes=0- means browser is asking server to return till whatever it can return ie. no end position is specified


https://bad-behavior.ioerror.us/support/faq/

7. Search


search

  1. file name
  2. description
  3. title
  4. cation

sorting

  1. size
  2. name
  3. date
  4. type

filter

  1. lang
  2. category
  3. user
  4. location

8. Links

  1. http://www.php.net/manual/en/features.file-upload.php
  2. https://www.owasp.org/index.ph[...]stricted_File_Upload
  3. https://www.acunetix.com/websi[...]upload-forms-threat/
  4. https://en.wikipedia.org/wiki/Media_type
  5. https://www.dokuwiki.org/images