File Upload and Media Handling


1. environment

php.ini

  • upload_max_filesize
  • post_max_size
  • max_input_time
  • max_execution_time
  • max_file_uploads
  • session.gc_maxlifetime

storage

  1. file system
    1. global
    2. perpage
  2. database

1.1. settings

options

  1. upload
    • [0, 1, Admins] - refactor, last value is actually just the allowed user group
  2. upload_max_size
  3. upload_quota
  4. upload_quota_per_user
  5. upload_banned_exts
  6. upload_images_only

added options

  1. check_mimetype
  2. img_create_thumbnail
  3. img_max_thumb_width
  4. upload_translit
  5. upload_translit_lower
  6. upload_allowed_exts

constants

  1. const THUMB_DIR = 'file/thumb';
  2. const THUMB_LOCAL_DIR = 'file/thumb_local';
  3. const UPLOAD_GLOBAL_DIR = 'file/global';
  4. const UPLOAD_LOCAL_DIR = 'file/perpage';

roles


permissions

  • $this->has_access('upload')

Anti-Hack

Download Flood Minutes:
The number of minutes to set flood control to.

Download Flood Control:

The number of downloads allowed in period of 'Download Flood Minutes'.

1.2. chmod

Write access for files folders.


  1. User can read, write and execute
  2. Group can read and execute
  3. World can read and execute

Execute privilege on directories is required to access files inside them. (AT&T)


The important thing is that the webserver has write access.


Thus if the files directory is owned by the webserver user ( e.g. www-data ) 755 would suffice.
If the webserver is not owner, but is in the group owning the directory, 775 would be better.
777 is the extreme case where all users on the system have full access. In most cases this would not be advisable.

1.3. MIME type and file extensions

https://en.wikipedia.org/wiki/Media_type
https://mimesniff.spec.whatwg.org/


Blacklisting File Extensions


Whitelisting File Extensions

# Allowed uploadable file extensions and mimetypes

$extension_map	= [
	// application
	'bz'		=> ['application', 'application/x-bzip'],
	'bz2'		=> ['application', 'application/x-bzip2'],
	'doc'		=> ['application', 'application/msword'],
	'exe'		=> ['application', 'application/octet-stream'],
	'latex'		=> ['application', 'application/x-latex'],
	'gtar'		=> ['application', 'application/x-gtar'],
	'gz'		=> ['application', 'application/x-gzip'],
	'gzip'		=> ['application', 'application/x-gzip'],
	'pdf'		=> ['application', 'application/pdf'],
	'ppt'		=> ['application', 'application/mspowerpoint'],
	'ps'		=> ['application', 'application/postscript'],
	'tar'		=> ['application', 'application/x-tar'],
	'tgz'		=> ['application', 'application/x-compressed'],
	'torrent'	=> ['application', 'application/x-bittorrent'],
	'xls'		=> ['application', 'application/excel'],
	'zip'		=> ['application', 'application/x-zip-compressed'],
	'7z'		=> ['application', 'application/x-7z-compressed'],

	'odc'		=> ['application', 'application/vnd.oasis.opendocument.chart'],
	'odb'		=> ['application', 'application/vnd.oasis.opendocument.database'],
	'odf'		=> ['application', 'application/vnd.oasis.opendocument.formula'],
	'odg'		=> ['application', 'application/vnd.oasis.opendocument.graphics'],
	'odi'		=> ['application', 'application/vnd.oasis.opendocument.image'],
	'odp'		=> ['application', 'application/vnd.oasis.opendocument.presentation'],
	'ods'		=> ['application', 'application/vnd.oasis.opendocument.spreadsheet'],
	'odt'		=> ['application', 'application/vnd.oasis.opendocument.text'],


	// (legacy)
	'pptx'		=> ['application', 'application/vnd.openxmlformats-officedocument.presentationml.presentation'],
	'xlsx'		=> ['application', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'],
	'docx'		=> ['application', 'application/vnd.openxmlformats-officedocument.wordprocessingml.document'],

	// audio
	'm4a'		=> ['audio', 'audio/mp4'],
	'mp3'		=> ['audio', 'audio/x-mpeg-3'],
	'ogg'		=> ['audio', 'audio/ogg'],
	'opus'		=> ['audio', 'audio/ogg'],

	// image
	'avif'		=> ['image', 'image/avif'],
	'gif'		=> ['image', 'image/gif'],
	'ico'		=> ['image', 'image/x-icon'],
	'jpe'		=> ['image', 'image/jpeg'],
	'jpeg'		=> ['image', 'image/jpeg'],
	'jpg'		=> ['image', 'image/jpeg'],
	'jxl'		=> ['image', 'image/jxl'],
	'png'		=> ['image', 'image/png'],
	'svg'		=> ['image', 'image/svg+xml'],
	'webp'		=> ['image', 'image/webp'],

	// text
	'conf'		=> ['text', 'text/plain'],
	'css'		=> ['text', 'text/css'],
	'htm'		=> ['text', 'text/html'],
	'html'		=> ['text', 'text/html'],
	'rtf'		=> ['text', 'text/richtext'],
	'txt'		=> ['text', 'text/plain'],
	'xml'		=> ['text', 'text/xml'],

	// video
	'mp4'		=> ['video', 'video/mp4'],
	'ogv'		=> ['video', 'video/ogg'],
	'webm'		=> ['video', 'video/webm'],
];

"Content-Type" Header Validation

1.4. HTML5


form

1.5. File APIs



Libs

1.6. Fetch API


Uploading files using 'fetch' and 'FormData'

1.7. PHP

POST method uploads


$_FILES

  • $_FILES['uploadedfile']['name']: The original name of the file on the client machine
  • $_FILES['uploadedfile']['type']: The mime type of the file
  • $_FILES['uploadedfile']['size']: The size of the file in bytes
  • $_FILES['uploadedfile']['tmp_name']: The temporary filename in which the uploaded file was stored on the server.

functions

  1. mime_content_type
  2. pathinfo
  3. hash_file

1.8. Image processing


  1. GD Graphics Library
  2. ImageMagick

check the availability

if (extension_loaded('gd'))
{
    print_r(gd_info());
}
else
{
    echo 'GD is not available.';
}

if (extension_loaded('imagick'))
{
    $imagick = new Imagick();
    print_r($imagick->queryFormats());
}
else
{
    echo 'ImageMagick is not available.';
}

2. WackoWiki file model

2.1. File name sanitization

All the control characters and Unicode ones should be removed from the filenames and their extensions without any exception. Also, the special characters such as ";", ":", ">", "<", "/" ,"\", additional ".", "*", "%", "$", and so on should be discarded as well. If it is applicable and there is no need to have Unicode characters, it is highly recommended to only accept Alpha-Numeric characters and only 1 dot as an input for the file name and the extension; in which the file name and also the extension should not be empty at all (regular expression: [a-zA-Z0-9]{1,200}\.[a-zA-Z0-9]{1,10}).


<?php

$sanitize_filename = function($name)
{
    // prepare for translit
    $name    = str_replace(['@', '%20', '+'], '_', $name);
    $name    = preg_replace('/[\r\n\t ]+/u', '_', $name);

    // remove multi full stop, spacing underscore and hyphen-minus
    $name    = preg_replace('/(-{2,})/u', '-', $name);
    $name    = preg_replace('/(_{2,})/u', '_', $name);
    $name    = preg_replace('/(\.{2,})/u', '.', $name);

    // remove consecutive occurrences (.- / -.)
    $name    = str_replace(['.-', '-.'], '', $name);
    $name    = utf8_trim($name, ' .-_');

    $name    = Ut::normalize($name);
    $name    = preg_replace('/[^' . self::PATTERN['ALPHANUM_P'] . '\.]/u', '', $name);

    // file name transliteration
    if ($this->db->upload_translit)
    {
        $t_name    = Ut::translit($name, $this->db->upload_translit_lower);
        $t_name    = preg_replace('/[\p{Z}]+/u', '_', $t_name);
    }
    else
    {
        $t_name    = $name;
    }

    return $t_name;
};

  1. global
    1. [file_name . extension]
    2. UPLOAD_GLOBAL_DIR
    3. direct file access
  2. local
    1. [@ . page_id . @ . file_name . extension]
    2. UPLOAD_LOCAL_DIR
    3. file access via file hadler
      1. checks page ACLs against user permissions

3. Database

  1. file_name
  2. author (user_id)
  3. file_description
  4. caption
  5. views
  6. uploaded_dt
  7. modified_dt
  8. credit
  9. exif
  10. context use (file_link)
  11. page_id
  12. categories
  13. access, viewing privacy
  14. file_lang
  15. file_size
  16. picture_w
  17. picture_h
  18. file_ext
  19. mime_type

4. Workflow


validation

  1. extension
  2. mime type
  3. size

header for file handler

"Content-Disposition: Attachment"
"X-Content-Type-Options: nosniff"

  1. https://developer.mozilla.org/[...]/Content-Disposition
  2. https://developer.mozilla.org/[...]Content-Type-Options

database

  1. mimetype (127+1+127 = 255)

https://stackoverflow.com/ques[...]n-storing-type-in-db


actions

  1. files
  2. upload

handler

  1. upload
  2. file
  3. attachments
  4. filemeta

functions

  • wacko -> link
  • wackoformatter

hacks


ideas

5. File Usage

Lotus Flower
Shinobazu Pond at Ueno Park at Ueno, Taito-ku in Tokyo
(Source:: Yoshikazu TAKADA /Some rights reserved)

The file syntax is processed in the link function. Not to mention that the link function requires a refactoring.


In R5.5 a cation field was added to the file table.

<figure>
  <img src="https://wackowiki.org/doc/file/global/lotus_flower.jpg" alt="Lotus Flower" title="Lotus Flower (105 KiB)" width="640" height="427">
  <figcaption>Shinobazu Pond at Ueno Park at Ueno, Taito-ku in Tokyo. Original by <a href="https://www.flickr.com/photos/yoshikazut/">Yoshikazu TAKADA</a></figcaption>
</figure>

Make the width of <figcaption> match the width of the <img> inside its <figure> tag

figure { display: table; }
figcaption { display: table-caption; caption-side:bottom; } 

The file can be linked directly (file handler) or to filemeta handler.


Arguments can be added via ? and &


file:/image.png?200x400&direct&caption

5.1. Supported Media


Media Format
audio m4a, mp3, ogg, opus
image avif, gif, jpg, jpe, jpeg, jxl, png, svg, webp
video mp4, ogv, webm

5.1.1. Audio

file:/audio.opus?caption

Johann Sebastian Bach - Well-Tempered Clavier, Book 1 - 01 Prelude No. 1 in C major, BWV 846
(Source:: Composer: Johann Sebastian Bach; Performer: Kimiko Ishizaka /Public domain)

https://developer.mozilla.org/[...]b/HTML/Element/audio

5.1.2. Video

file:/video.webm?caption

Sandhill Cranes
(Source:: NPS / Dale Bohlke /Public domain)

https://developer.mozilla.org/[...]b/HTML/Element/video
https://developer.mozilla.org/[...]itles_to_HTML5_video

5.2. Embedding

file:image.png
file:audio.opus
file:video.mp4

5.3. Media Parameters


img audio video notes
linking
direct x o o
nolink x o o
linkonly x o o
meta x o o
alignment
right x x x
left x x x
center x x x
resizing
400x200 x o x
600 x x x
others
caption x x x
clear x x x

5.3.1. Alignment

file:image.png?right


file:image.png?right&clear

5.3.2. Resizing

file:image.png?20x50


Lotus Flower
Shinobazu Pond at Ueno Park at Ueno, Taito-ku in Tokyo
(Source:: Yoshikazu TAKADA /Some rights reserved)

5.3.3. Linking

file:image.png?direct
file:image.png?nolink
file:image.png?linkonly


By default, clicking on an image should brings up the filemeta handler page.


How we manage to link an image to another page with the file: syntax?

  • ((file:/image.png some text))
  • ((/Page/Link file:/image.png))

5.3.4. Caching

file:image.png?nocache

5.3.5. Others

file:image.png?caption


+ audio and video options

6. Bad Behavior strict mode issue

bb_settings.conf

strict = true

status_key: 7ad04a8a


HTTP load failed with status 400. Load of media resource https://wackowiki.org/doc/file/global/41265046264_720p.mp4 failed.
HTTP load failed with status 400. Load of media resource https://wackowiki.org/doc/file/global/kimiko_ishizaka___bach___well_tempered_clavier_book_1___01_prelude_no._1_in_c_major_bwv_846.ogg failed.


[400] The automated program you are using is not permitted to access this server. Please use a different program or a standard Web browser.
-> Prohibited header 'Range' present


Range: bytes=0-
Referer: https://wackowiki.org/doc/Dev/Release/R5.5/FileUpload
Accept-Language: en-US,en;q=0.5
Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0
Host: wackowiki.org
Mod-Rewrite: on
Mod-Env: on

Suppose you have a video or audio of 4M bytes, when your browser request for video the first time it will send headers like this

Host:localhost
Range:bytes=0-

Range header bytes=0- means browser is asking server to return till whatever it can return ie. no end position is specified


https://bad-behavior.ioerror.us/support/faq/

7. Search


search

  1. file name
  2. description
  3. title
  4. cation

sorting

  1. size
  2. name
  3. date
  4. type

filter

  1. lang
  2. category
  3. user
  4. location

8. Links

  1. http://www.php.net/manual/en/features.file-upload.php
  2. https://www.owasp.org/index.ph[...]stricted_File_Upload
  3. https://www.acunetix.com/websi[...]upload-forms-threat/
  4. https://en.wikipedia.org/wiki/Media_type
  5. https://www.dokuwiki.org/images